Feat: flat pivot table

[djbarnwal]

Addresses #6711

Remaining Items -

• Ability to re-arrange columns to mix measures and dimensions (deferred to next PR due to scale of change across codebase) feat: allow mixed ordering in flat pivot table #6784

Checklist:

• Covered by tests
• Ran it and it works as intended
• Reviewed the diff before requesting a review
• Checked for unhandled edge cases
• Linked the issues it closes
• Checked if the docs need to be updated (Updates needed)
• Intend to cherry-pick into the release branch
• I'm proud of this work!

[djbarnwal]

@AdityaHegde tagging you for review of URL and relevant proto changes, @ericpgreen2 for other files.

[AdityaHegde]

You also need to update dashboard.proto and add proto conversions. Bookmarks and public urls still use the old proto.

[AdityaHegde]

Approving from URL params POV. PivotRowJoinType is already in dashboard.proto so it is not in the diffs.

[ericpgreen2]

Looks pretty good!

In general, I'm wary of many isFlat conditionals that now permeate the pivot code. There's a lot of nested conditionals. As the pivot/table configuration grows, we should think about how we can avoid even more conditionals (though I don't have a concrete proposal at the moment).

[ericpgreen2]

Why are these the rowDimensions, not the colDimensions? Given the pills are labeled as "Columns":

[djbarnwal]

This is because the way implementation is set up for Pivot tables. Column dimension internally are treated as values on which measures would be split and hence have a very different way of working. For flat table we need just a basic query so it made sense to use rowDimensions as a proxy for implementing it. I have added a comment for this in pivot-data-config.

[ericpgreen2]

Note: there's a lot of nesting here

[ericpgreen2]

What's this mean?

[djbarnwal]

There are multiple stages in a pivot construction. assembled: true signifies that all the stages have been completed.

[djbarnwal]

Looks pretty good!

In general, I'm wary of many isFlat conditionals that now permeate the pivot code. There's a lot of nested conditionals. As the pivot/table configuration grows, we should think about how we can avoid even more conditionals (though I don't have a concrete proposal at the moment).

Have made the proto changes to rename rowJoinType to tableMode.

FWith regards to nesting, the other option is to create a separate data store for flat table. Though, the implementation paths for both are very similar with flat table needed few steps so a lot of the conditionals are used to skip those steps. We can create a different data store for flat but it would mean duplicating a lot of existing nested table code.

[ericpgreen2]

With regards to nesting, the other option is to create a separate data store for flat table. Though, the implementation paths for both are very similar with flat table needed few steps so a lot of the conditionals are used to skip those steps. We can create a different data store for flat but it would mean duplicating a lot of existing nested table code.

I discussed it with Claude-3.7-sonnet in Cursor, and I got this high-level recommendation:

// Create specialized stores
function createFlatPivotDataStore(ctx, config) {
  // Flat-specific implementation
}

function createNestedPivotDataStore(ctx, config) {
  // Nested-specific implementation
}

// Facade that delegates to the appropriate store
export function createPivotDataStore(ctx, configStore) {
  return derived(configStore, (config, set) => {
    const storeFactory = config.isFlat 
      ? createFlatPivotDataStore 
      : createNestedPivotDataStore;
    
    return storeFactory(ctx, config).subscribe(set);
  });
}

For any of the common processing steps, we could of course use a common utility function. But I know this might be a largish refactor of what you have, so I will defer to you on what you think is best.

[ericpgreen2]

Nit: Going forward, it'd be nice to break out the "run through" tests into smaller pieces. Then, when they fail, we can more easily see which sub-feature is the culprit. In this case, I could see a test dedicated to the Flat Table.