Simultaneous Sorting/Removing data from a List

Currently I have a list of integers. This list contains index values that point to "active" objects in another, much larger list. If the smaller list of "active" values becomes too large, it triggers a loop that iterates through the small list and removes values that have become inactive. Currently, it removes them by simply ignoring the inactive values and adding them to a second list (and when the second list gets full again the same process is repeated, placing them back into the first list and so on).

After this trigger occurs, the list is then sorted using a Quicksort implementation. This is all fine and dandy.

-------Question---------

However, I see a potential gain of speed. I am imagining combining the removal of inactive values while the sorting is taking place. Unfortunately, I cannot find a way to implement quicksort in this way. Simply because the quicksort works with pivots, which means if values are removed from the list, the pivot will eventually try to access a slot in the list that does not exist, etc etc.. (unless I'm just thinking about it wrong).

So, any ideas on how to combine the two operations? I can't seem to find any sorting algorithms as fast as quicksort that could handle this, or perhaps I'm just not seeing how to implement it into a quicksort... any hints are appreciated!

Code for better understanding of whats currently going on: (Current Conditions: values can range from 0 to 2 million, no 2 values are the same, and in general they are mostly sorted, since they are sorted every so often)

if (deactive > 50000)//if the number of inactive coordinates is greater than 50k
{
    for (int i = 0; i < activeCoords1.Count; i++)
    {
         if (largeArray[activeCoords[i]].active == true)//if coordinate is active, readd to list
         {
             activeCoords2.Add(activeCoords1[i]);
         }
    }
    //clears the old list for future use
    activeCoords1.Clear();
    deactive = 0;
    //sorts the new list
    Quicksort(activeCoords2, 0, activeCoords2.Count() - 1);
 }    

    static void Quicksort(List<int> elements, int left, int right)
    {
        int i = left, j = right;
        int pivot = elements[(left + right) / 2];

        while (i <= j)
        {
            // p < pivot
            while (elements[i].CompareTo(pivot) < 0)
            {
                i++;
            }

            while (elements[j].CompareTo(pivot) > 0)
            {
                j--;
            }

            if (i <= j)
            {
                // Swap
                int tmp = elements[i];
                elements[i] = elements[j];
                elements[j] = tmp;

                i++;
                j--;
            }
        }

        // Recursive calls
        if (left < j)
        {
            Quicksort(elements, elements, left, j);
        }

        if (i < right)
        {
            Quicksort(elements, elements, i, right);
        }
    }