Thread: [pgAdmin4][Patch] - Encoding Fixes

Hi,

Please find the attached patch to support all the supported encodings by PostgreSQL.

Thanks,

Khushboo

On Thu, Feb 28, 2019 at 10:53 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

On Wed, Feb 27, 2019 at 4:57 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Feb 26, 2019 at 9:01 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to fix the encoding related issues.

#3992 - View data causes errors with DB encoding EUC_JIS_2004

#3982 - Encoding Problem with PGAdmin 4.2

#3911 - Data output does not display Arabic in WIN1256 encoding database

- The CSV download for the same encoding has also been fixed.

I have tested the patch on Python 2.7, 3.5, 3.6 and 3.7.

You've added a couple of new encodings, but it seems to me that we should just add everything that PostgreSQL supports. For example, we now have win1256, but what about 866, 874, 1250, 1251 etc?

The full list is at https://www.postgresql.org/docs/11/multibyte.html

Any reason not to do that?

I thought to fix the reported bugs first and after that, I was about to add a new ticket to add the remaining supported encodings.

But as per you, I will add the remaining encodings along with these fixes.

 

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Thanks, applied.

On Fri, Mar 1, 2019 at 7:06 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to support all the supported encodings by PostgreSQL.

Thanks,

Khushboo

On Thu, Feb 28, 2019 at 10:53 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

On Wed, Feb 27, 2019 at 4:57 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Feb 26, 2019 at 9:01 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to fix the encoding related issues.

#3992 - View data causes errors with DB encoding EUC_JIS_2004

#3982 - Encoding Problem with PGAdmin 4.2

#3911 - Data output does not display Arabic in WIN1256 encoding database

- The CSV download for the same encoding has also been fixed.

I have tested the patch on Python 2.7, 3.5, 3.6 and 3.7.

You've added a couple of new encodings, but it seems to me that we should just add everything that PostgreSQL supports. For example, we now have win1256, but what about 866, 874, 1250, 1251 etc?

The full list is at https://www.postgresql.org/docs/11/multibyte.html

Any reason not to do that?

I thought to fix the reported bugs first and after that, I was about to add a new ticket to add the remaining supported encodings.

But as per you, I will add the remaining encodings along with these fixes.

 

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi

Unfortunately this has introduced a regression:

- Start pgAdmin from the command line or PyCharms.

- Connect to a server, open a query tool and run a query (SELECT 1 will do).

- Stop and restart the pgAdmin server.

- Execute the query again in the client.

The error "'Connection' object has no attribute 'python_encoding'" is returned in the messages tab and the following exception is given on the pgAdmin server:

2019-03-01 16:07:26,293: ERROR pgadmin: 'Connection' object has no attribute 'python_encoding'

Traceback (most recent call last):

  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/flask/app.py", line 1612, in full_dispatch_request

    rv = self.dispatch_request()

  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/flask/app.py", line 1598, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/flask_login.py", line 792, in decorated_view

    return func(*args, **kwargs)

  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 321, in start_query_tool

    sql, trans_id, session, connect

  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/utils/start_running_query.py", line 87, in execute

    transaction_object

  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/utils/start_running_query.py", line 136, in __execute_query

    status, result = conn.execute_async(sql)

  File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 939, in execute_async

    encoding = self.python_encoding

AttributeError: 'Connection' object has no attribute 'python_encoding'

I'm guessing there's a codepath in which connection objects are created without the python_encoding attribute.

On Fri, Mar 1, 2019 at 1:54 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks, applied.

On Fri, Mar 1, 2019 at 7:06 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to support all the supported encodings by PostgreSQL.

Thanks,

Khushboo

On Thu, Feb 28, 2019 at 10:53 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

On Wed, Feb 27, 2019 at 4:57 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Feb 26, 2019 at 9:01 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to fix the encoding related issues.

#3992 - View data causes errors with DB encoding EUC_JIS_2004

#3982 - Encoding Problem with PGAdmin 4.2

#3911 - Data output does not display Arabic in WIN1256 encoding database

- The CSV download for the same encoding has also been fixed.

I have tested the patch on Python 2.7, 3.5, 3.6 and 3.7.

You've added a couple of new encodings, but it seems to me that we should just add everything that PostgreSQL supports. For example, we now have win1256, but what about 866, 874, 1250, 1251 etc?

The full list is at https://www.postgresql.org/docs/11/multibyte.html

Any reason not to do that?

I thought to fix the reported bugs first and after that, I was about to add a new ticket to add the remaining supported encodings.

But as per you, I will add the remaining encodings along with these fixes.

 

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

Please find the attached patch for the regression fix.

Thanks,

Khushboo

On Fri, Mar 1, 2019 at 9:41 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

Unfortunately this has introduced a regression:

- Start pgAdmin from the command line or PyCharms.

- Connect to a server, open a query tool and run a query (SELECT 1 will do).

- Stop and restart the pgAdmin server.

- Execute the query again in the client.

The error "'Connection' object has no attribute 'python_encoding'" is returned in the messages tab and the following exception is given on the pgAdmin server:

2019-03-01 16:07:26,293: ERROR pgadmin: 'Connection' object has no attribute 'python_encoding'

Traceback (most recent call last):

  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/flask/app.py", line 1612, in full_dispatch_request

    rv = self.dispatch_request()

  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/flask/app.py", line 1598, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/flask_login.py", line 792, in decorated_view

    return func(*args, **kwargs)

  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 321, in start_query_tool

    sql, trans_id, session, connect

  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/utils/start_running_query.py", line 87, in execute

    transaction_object

  File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/utils/start_running_query.py", line 136, in __execute_query

    status, result = conn.execute_async(sql)

  File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py", line 939, in execute_async

    encoding = self.python_encoding

AttributeError: 'Connection' object has no attribute 'python_encoding'

I'm guessing there's a codepath in which connection objects are created without the python_encoding attribute.

On Fri, Mar 1, 2019 at 1:54 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks, applied.

On Fri, Mar 1, 2019 at 7:06 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to support all the supported encodings by PostgreSQL.

Thanks,

Khushboo

On Thu, Feb 28, 2019 at 10:53 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

On Wed, Feb 27, 2019 at 4:57 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Feb 26, 2019 at 9:01 AM Khushboo Vashi <khushboo.vashi@enterprisedb.com> wrote:

Hi,

Please find the attached patch to fix the encoding related issues.

#3992 - View data causes errors with DB encoding EUC_JIS_2004

#3982 - Encoding Problem with PGAdmin 4.2

#3911 - Data output does not display Arabic in WIN1256 encoding database

- The CSV download for the same encoding has also been fixed.

I have tested the patch on Python 2.7, 3.5, 3.6 and 3.7.

You've added a couple of new encodings, but it seems to me that we should just add everything that PostgreSQL supports. For example, we now have win1256, but what about 866, 874, 1250, 1251 etc?

The full list is at https://www.postgresql.org/docs/11/multibyte.html

Any reason not to do that?

I thought to fix the reported bugs first and after that, I was about to add a new ticket to add the remaining supported encodings.

But as per you, I will add the remaining encodings along with these fixes.

 

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company