Re: PG 9.3 materialized view VS Views, indexes, shared memory - Mailing list pgsql-performance
| From | Nicolas Paris |
|---|---|
| Subject | Re: PG 9.3 materialized view VS Views, indexes, shared memory |
| Date | |
| Msg-id | CA+ssMOTCDWk_1BbeaqcepfsRa_KhkSx979nfUxjK3Eu3GZzzew@mail.gmail.com Whole thread Raw |
| In response to | Re: PG 9.3 materialized view VS Views, indexes, shared memory (Jim Nasby <Jim.Nasby@BlueTreble.com>) |
| Responses |
Re: PG 9.3 materialized view VS Views, indexes, shared
memory
|
| List | pgsql-performance |
If you have that many different views I doubt you want that many indexes anyway.
It's a datawarehouse, then each view is used by many user for each query.
Those views must be subset of the huge material table. All indexes are needed
Have you tried just hitting the base table and indexes directly, either through plain views or just direct SQL?
I have tried each. The performances are worst querying on a subset (the views) than querying on whole huge table when using the huge indexes
=> this is the solution I am implementing. (800 is not true, but in 10 years it maybe will be)
Actually, I have added a boolean column on the huge table for each views. This is the way each view is a subset of huge table (Create View as Select * FROM hugeTable WHERE columnX is true --etc 800 times). Then I create 800partials indexes on that column(create index...WHERE columnX is TRUE), for each view.
This works great as the query planer chooses the partials indexes when querying the little subset of the terrific table (potential 20bilion rows)
This is better than material views for some reasons :
- saves places on hard drive (columnX is boolean +same indexes - data for MatViews)
- saves time generating materialised views
This is quite more complicated because in the project, the number of view is increasing, and dynamic then :
- then adding new mat views is simple
- adding new views => adding new column on the huge table. It can take long time to update boolean for each tuple. Then I need to truncate/bulk load all data each time I add a new View. Other problem is dynamic number column table was a bit tricky to implement in an ETL soft such Talend, but the benefits are I hope great.
Nicolas PARIS
2015-03-06 2:40 GMT+01:00 Jim Nasby <Jim.Nasby@bluetreble.com>:
On 2/20/15 12:09 PM, Nicolas Paris wrote:Well it seems that max query size for CREATE INDEX is 8160 character in
my 9.3 postgresql version.
Then the only solution see is to add a new boolean field : huge_table.view1
and change predicat to "WHERE view1=1 "
But I may have 800 views.. adding 800 new fields indexed to the huge
table is actually not a good idea. Too bad
Any idea to solve that partial view limitation?
If you have that many different views I doubt you want that many indexes anyway.
Have you tried just hitting the base table and indexes directly, either through plain views or just direct SQL?
Also, how frequently does data change in the huge table? This sounds like a case where the visibility map could make a huge difference.
By the way, if all the Mat Views are in one schema that's already in the search path, a very easy way to test this would be to create an equivalent set of regular views in a different schema (which you can probably do programmatically via pg_get_viewdef()) and then change the search_path to put the new schema before the old.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com
pgsql-performance by date: