LINQ in Depth: Iterators, Joins & Performance

In LINQ you learned to filter, sort, group and join collections. That is enough to read most code, but interviews and real projects go further: Why is a query lazy? How do I write my own operator? Why does my EF Core query load the whole table? Why is this report slow with 50,000 rows?

This lesson opens up LINQ. You will see how yield return iterators power every operator, build your own operator step by step, meet the operators that flatten and compare sequences, and learn the performance traps that experienced developers watch for.

How LINQ really works: iterators

Every LINQ method such as Where or Select is an extension method on IEnumerable<T> that returns another IEnumerable<T>. Most are written as iterators: methods that use yield return to hand out one item at a time. The method pauses after each yield return and continues from the same place when the next item is requested.

Watch the order of the printed lines carefully:

var query = Numbers()
    .Where(n => n % 2 == 0)
    .Select(n => n * 10);

Console.WriteLine("Query built");

foreach (int x in query)
{
    Console.WriteLine($"Got {x}");
}

IEnumerable<int> Numbers()
{
    for (int i = 1; i <= 4; i++)
    {
        Console.WriteLine($"  yield {i}");
        yield return i;
    }
}

Query built
  yield 1
  yield 2
Got 20
  yield 3
  yield 4
Got 40

Nothing runs when the query is built. When foreach asks for the first item, Select asks Where, which asks Numbers. The number 1 fails the filter, so Where asks again; 2 passes and travels all the way to the loop as 20. Items flow through the chain one at a time, which is why LINQ can process a file of a million lines without loading it all into memory.

Walkthrough: write your own operator

Suppose a cricket app needs every third ball for a slow-motion replay. There is no built-in EveryNth, so you write one.

Step 1: an extension method with an iterator

public static class SequenceExtensions
{
    public static IEnumerable<T> EveryNth<T>(this IEnumerable<T> source, int n)
    {
        int index = 0;
        foreach (T item in source)
        {
            if (index % n == 0)
            {
                yield return item;
            }
            index++;
        }
    }
}

It works, but it has a hidden flaw. Because the whole method is an iterator, none of its code runs until someone enumerates the result, including any argument checks you add at the top.

Step 2: validate eagerly, iterate lazily

The standard fix splits the method in two: the outer method checks arguments immediately, and a local iterator function does the lazy work.

public static class SequenceExtensions
{
    public static IEnumerable<T> EveryNth<T>(this IEnumerable<T> source, int n)
    {
        ArgumentNullException.ThrowIfNull(source);
        ArgumentOutOfRangeException.ThrowIfLessThan(n, 1);
        return Iterate();

        IEnumerable<T> Iterate()
        {
            int index = 0;
            foreach (T item in source)
            {
                if (index % n == 0)
                {
                    yield return item;
                }
                index++;
            }
        }
    }
}

Now balls.EveryNth(0) throws ArgumentOutOfRangeException on the line where it is called, not later in some unrelated loop. The .NET team writes its own operators the same way.

Step 3: use it like any LINQ method

Console.WriteLine(string.Join(", ", Enumerable.Range(1, 10).EveryNth(3)));

string[] balls = ["0", "4", "1", "W", "6", "2", "0"];
Console.WriteLine(string.Join(" ", balls.EveryNth(3).Select(b => $"[{b}]")));

1, 4, 7, 10
[0] [W] [0]

Your operator chains with Select, Where and the rest because it takes and returns IEnumerable<T>. Items at positions 0, 3 and 6 are picked.

Flattening, pairing and comparing sequences

SelectMany flattens: each item produces a sequence, and the results are joined into one. Zip walks two sequences side by side. The ...By set operators (added in .NET 6) compare items by a key you choose.

List<Order> orders =
[
    new("Aarav", ["Pen", "Notebook"]),
    new("Priya", ["Notebook", "Bag", "Pen"]),
    new("Kabir", ["Geometry box"])
];

Console.WriteLine(string.Join(", ", orders.SelectMany(o => o.Items).Distinct()));

var lines = orders.SelectMany(o => o.Items, (o, item) => $"{o.Customer}:{item}");
Console.WriteLine(string.Join(" ", lines.Take(3)));

string[] days = ["Mon", "Tue", "Wed"];
int[] sales = [1200, 950, 1430];
foreach (var (day, amount) in days.Zip(sales))
{
    Console.WriteLine($"{day} {amount}");
}

List<Student> registered = [new("S1", "Meera"), new("S2", "Rohan"), new("S3", "Isha")];
List<Student> paid = [new("S2", "Rohan")];
var unpaid = registered.ExceptBy(paid.Select(p => p.Id), s => s.Id);
Console.WriteLine("Unpaid: " + string.Join(", ", unpaid.Select(s => s.Name)));

record Order(string Customer, List<string> Items);
record Student(string Id, string Name);

Pen, Notebook, Bag, Geometry box
Aarav:Pen Aarav:Notebook Priya:Notebook
Mon 1200
Tue 950
Wed 1430
Unpaid: Meera, Isha

• The second form of SelectMany keeps the parent: it gives you both the order and the item, perfect for building order lines.
• Zip stops at the end of the shorter sequence, so mismatched lengths do not crash, but they can silently drop data.
• DistinctBy, UnionBy, IntersectBy and ExceptBy compare only the key. Here the office compares student IDs, not whole records.

Lookups, dictionaries and Aggregate

ToDictionary and ToLookup run the query immediately and build a structure for fast lookups by key. A lookup is like a dictionary whose values are groups. Aggregate folds a whole sequence into one value with your own rule.

List<Player> squad =
[
    new("Rohan", "Pune"), new("Farhan", "Lucknow"), new("Ananya", "Pune")
];

ILookup<string, Player> byCity = squad.ToLookup(p => p.City);
Console.WriteLine(string.Join(", ", byCity["Pune"].Select(p => p.Name)));
Console.WriteLine(byCity["Kochi"].Count());

Dictionary<string, string> cityOf = squad.ToDictionary(p => p.Name, p => p.City);
Console.WriteLine(cityOf["Farhan"]);

decimal[] yearlyRates = [7m, 7.5m];
decimal deposit = yearlyRates.Aggregate(10000m, (amount, rate) => amount * (1 + rate / 100));
Console.WriteLine($"After 2 years: {deposit:F2}");

record Player(string Name, string City);

Rohan, Ananya
0
Lucknow
After 2 years: 11502.50

Asking a lookup for a missing key returns an empty sequence, not an exception, which makes it safer than a dictionary for grouping. ToDictionary throws ArgumentException if two items have the same key, so use it only on unique keys. Aggregate starts from the seed (10,000) and applies each year's rate in turn: 10,000 x 1.07 x 1.075 = 11,502.50.

IEnumerable vs IQueryable

LINQ over a List uses IEnumerable<T>: your lambdas are compiled C# code that runs in your program. LINQ over an EF Core DbSet uses IQueryable<T>: your lambdas become expression trees, data that describes the code, and the provider translates them into SQL.

// IQueryable: Where and OrderBy become SQL; only matching rows come back
var recent = await db.Books
    .Where(b => b.Year >= 2000)
    .OrderBy(b => b.Title)
    .ToListAsync();

// Mistake: AsEnumerable switches to in-memory LINQ, so every row is loaded first
var slow = db.Books.AsEnumerable().Where(b => b.Year >= 2000).ToList();

Two rules follow. First, keep filtering on the IQueryable side and call ToListAsync last, so the database does the work. Second, only use things the provider can translate. A call to your own C# method inside Where cannot become SQL, and EF Core throws an InvalidOperationException saying the LINQ expression could not be translated. The Entity Framework Core lesson shows these queries in context.

Performance traps

Most slow LINQ code comes from a few habits:

int calls = 0;
IEnumerable<int> squares = Enumerable.Range(1, 3).Select(n => { calls++; return n * n; });

Console.WriteLine(squares.Sum());
Console.WriteLine(squares.Max());
Console.WriteLine($"Selector ran {calls} times");

calls = 0;
List<int> cached = squares.ToList();
Console.WriteLine(cached.Sum() + cached.Max());
Console.WriteLine($"Selector ran {calls} times");

14
9
Selector ran 6 times
23
Selector ran 3 times

• Multiple enumeration. A deferred query runs again every time you use it. With a database or a file, that means repeated I/O. Call ToList() once when you need the results more than once.
• Searching inside a loop. orders.Where(o => vipNames.Contains(o.Customer)) with a List<string> checks every name for every order. Put the names in a HashSet<string> (or use ToLookup) and each check becomes almost instant.
• Count() > 0 instead of Any(). Any stops at the first item.
• Sorting just to take one item. OrderBy(...).First() sorts everything; MinBy and MaxBy scan once.

Common mistakes

• Putting argument checks inside an iterator. They run late, far from the bad call. Use the outer method plus local iterator pattern.
• Calling AsEnumerable() or ToList() too early on a database query. Everything after it runs in memory on the full table.
• Using ToDictionary on keys that are not unique. It throws; use ToLookup or GroupBy.
• Changing a collection while a query is iterating it. List<T> throws InvalidOperationException: Collection was modified. Materialise with ToList() first.

Key points

• LINQ operators are iterators: items flow through the chain one at a time when you enumerate.
• Write custom operators as extension methods; validate arguments eagerly and use a local iterator function.
• SelectMany flattens, Zip pairs, and DistinctBy, ExceptBy, IntersectBy, UnionBy compare by key.
• ToLookup, ToDictionary and Aggregate run immediately; a lookup returns an empty group for a missing key.
• IQueryable queries are translated to SQL, so filter before materialising and avoid untranslatable calls.
• Avoid multiple enumeration and nested searches; measure before optimising.

foreach (List<int> batch in Enumerable.Range(1, 7).Batch(3))
{
    Console.WriteLine(string.Join(",", batch));
}

public static class BatchExtensions
{
    public static IEnumerable<List<T>> Batch<T>(this IEnumerable<T> source, int size)
    {
        ArgumentNullException.ThrowIfNull(source);
        ArgumentOutOfRangeException.ThrowIfLessThan(size, 1);
        return Iterate();

        IEnumerable<List<T>> Iterate()
        {
            List<T> current = new(size);
            foreach (T item in source)
            {
                current.Add(item);
                if (current.Count == size)
                {
                    yield return current;
                    current = new(size);
                }
            }
            if (current.Count > 0)
            {
                yield return current;
            }
        }
    }
}

1,2,3
4,5,6
7