First, create a list of all triangles (subsets of 3 points). Then you make a pairwise comparison of each of two triangles T_i and T_j: either T_i is inside T_j, T_j is inside T_i or none of the two lies inside of the other.
Interpret this as a directed, acyclic graph: the triangles are the vertices of the graph, and each relationship "T_i is inside T_j" defines a directed edge from T_i to T_j. Now finding of a maximum sequence of triangles is just the longest path problem for such kind of graphs. And as you can read in the Wikipedia article, there exist linear time algorithms for solving this problem.
"Linear time" here means "linear to the number of edges" (the number of triangle pairs where one is inside the other). For n points, you have to consider
t := C(n,3)=n*(n-1)*(n-2)/6
triangles (see binomial coefficient), and thus a maximum of
C(t,2) = t*(t-1)/2
triangle pairs, which can be a huge number with increasing n - its a polynomial of order O(n^6). But since most triangle pairs are not expected to of the type "where one is inside the other", I would expect the real world computational effort to be much smaller. For n below 100 it should be no problem to find a solution directly.
For n >=1.000, this approach most probably won't be fast enough to give you a result in a reasonable amount of time. Thus, better try to solve it with an approximating algorithm like the one suggested by @KonradMorawski, or (easier to implement) simulated annealing. The "simulated annealing" will need a "small modification" step, which may be implemented by removing one or two randomly choosen triangles from your "current solution", and then add triangles again by some kind of "greedy algorithm" with a bit of randomness. You will surely need to experiment with these details to see what works best for your specific problem. As a free resource about this topics, here you find an e-book dealing with different global optimization techniques.