We have an Avro file that has an ID, AttributeName and AttributeValue… Very narrow, but very deep.
475MM unique ID’s
20 unique AttributeName’s (Avro file is partitioned on AttributeName)
What are the best technologies and syntax to pivot a huge file like this?**
Here is an example: