I’m building a geoprocessing service that queries an SDE database (running on postgres) and returns results. I do a lot of development by calling the script from a toolbox in ArcGIS desktop 10.3.

For some reason when I call the script below in ArcGIS desktop, it executes successfully the first time, then fails every time after that with ‘NoneType’ object is not callable
on line 12 as I try to iterate over inAllCornerPointsList. If I close ArcMap and call it, it works the first time, then fails.

import psycopg2

conn = psycopg2.connect(host='localhost', database ='sde', user='sde', password='somepassword')
curs = conn.cursor()

sqlStatement = """SELECT * FROM sde.gly_ula_corners"""

curs.execute(sqlStatement)
inAllCornerPointsList = curs.fetchall()

for cornerPoint in inAllCornerPointsList:
    print cornerPoint

curs.close()
conn.close()

print 'Complete'

To further confuse things, if I select * from a different table (sde.countyboundaries), my script executes successfully every time.

The geometry type for sde.gly_ula_corners is shape st_point. The geometry type for sde.countyboundaries is shape st_geometry. This is the only difference that jumps out at me right away– one is a point fc, and one is a polygon.

Any idea what might be causing this? Is there some weird memory environment at work? Calling the script multiple times from the command line or pythonwin does not fail.

I’m trying to write a munin plugin to graph DB sizes. Alongside using pg_database_size I want to graph the components thereof as well.

So far, I’ve come up with the following:

SELECT
    SUM(pg_relation_size(oid, 'main')) AS main_size,
    SUM(pg_relation_size(oid, 'vm')) AS vm_size,
    SUM(pg_relation_size(oid, 'fsm')) AS fsm_size,
    SUM(
        CASE reltoastrelid
        WHEN 0 THEN 0
        ELSE pg_total_relation_size(reltoastrelid)
        END
    ) AS toast_size,
    SUM(pg_indexes_size(oid)) AS indexes_size
    FROM pg_class
    WHERE reltype != 0 -- 0=indices, covered by pg_indexes_size

However, summing up those 5 values returns me something which is not the same as the result of pg_database_size. The difference seems to be less significant for larger DBs.

Example on a larger DB:

┌──────────┬────────┬─────────┬─────────┬──────────┬───────────────┬──────────────────┬─────────┐
│   main   │   vm   │   fsm   │  toast  │ indexes  │ sum_of_values │ pg_database_size │  diff   │
├──────────┼────────┼─────────┼─────────┼──────────┼───────────────┼──────────────────┼─────────┤
│ 72441856 │ 753664 │ 2392064 │ 4677632 │ 41377792 │ 116 MB        │ 111 MB           │ 5222 kB │
└──────────┴────────┴─────────┴─────────┴──────────┴───────────────┴──────────────────┴─────────┘
(1 row)

Example on a smaller DB:

┌─────────┬────────┬─────────┬────────┬─────────┬───────────────┬──────────────────┬─────────┐
│  main   │   vm   │   fsm   │ toast  │ indexes │ sum_of_values │ pg_database_size │  diff   │
├─────────┼────────┼─────────┼────────┼─────────┼───────────────┼──────────────────┼─────────┤
│ 2809856 │ 385024 │ 1351680 │ 557056 │ 2924544 │ 7840 kB       │ 6642 kB          │ 1198 kB │
└─────────┴────────┴─────────┴────────┴─────────┴───────────────┴──────────────────┴─────────┘
(1 row)

What am I missing?

Maybe related, maybe not: I’m shocked to see the index size. They are HUGE. Is something in my query wrong?

Here is a script I used to inspect the different values:

SELECT
    SUM(pg_relation_size(oid, 'main')) AS main,
    SUM(pg_relation_size(oid, 'vm')) AS vm,
    SUM(pg_relation_size(oid, 'fsm')) AS fsm,
    SUM(
        CASE reltoastrelid
        WHEN 0 THEN 0
        ELSE pg_total_relation_size(reltoastrelid)
        END
    ) AS toast,
    SUM(pg_indexes_size(oid)) AS indexes,
    pg_size_pretty(
        SUM(pg_relation_size(oid, 'main'))::bigint +
        SUM(pg_relation_size(oid, 'vm'))::bigint +
        SUM(pg_relation_size(oid, 'fsm'))::bigint +
        SUM(pg_indexes_size(oid))::bigint +
        SUM(
            CASE reltoastrelid
            WHEN 0 THEN 0
            ELSE pg_total_relation_size(reltoastrelid)
            END
        )::bigint
    ) AS sum_of_values,
    pg_size_pretty(pg_database_size(current_database())) AS pg_database_size,

    pg_size_pretty(
        SUM(pg_relation_size(oid, 'main'))::bigint +
        SUM(pg_relation_size(oid, 'vm'))::bigint +
        SUM(pg_relation_size(oid, 'fsm'))::bigint +
        SUM(pg_indexes_size(oid))::bigint +
        SUM(
            CASE reltoastrelid
            WHEN 0 THEN 0
            ELSE pg_total_relation_size(reltoastrelid)
            END
        )::bigint - pg_database_size(current_database())::bigint
    ) AS diff

FROM pg_class
WHERE reltype != 0;

In Postgres 9.5, I was surprised to see the result seen below while experimenting with year 0001 (no year zero 0000).

Offset of -07:52:58?

Some example code. Note that I mixed use of TIMESTAMP WITH TIME ZONE and TIMESTAMP WITHOUT TIME ZONE, so read carefully.

SET TIME ZONE 'America/Los_Angeles' ;

SELECT (TIMESTAMP WITH TIME ZONE '2015-01-01 00:00:00.0', 
        TIMESTAMP WITH TIME ZONE '0001-01-01 00:00:00.0Z', 
        TIMESTAMP WITHOUT TIME ZONE '0001-01-01 00:00:00.0Z') ;

("2015-01-01 00:00:00-08","0001-12-31 16:07:02-07:52:58 BC","0001-01-01 00:00:00")

I am surprised by that second value: 0001-12-31 16:07:02-07:52:58 BC. I understand that we must go backwards eight hours as America/Los_Angeles is eight hours behind UTC with an offset of -08:00. But instead of -08:00 the offset is -07:52:58. Why?

No Problem Under UTC

No such issue when entering data under UTC.

SET TIME ZONE 'UTC' ;

SELECT (TIMESTAMP WITH TIME ZONE '2015-01-01 00:00:00.0',  
        TIMESTAMP WITH TIME ZONE '0001-01-01 00:00:00.0Z', 
        TIMESTAMP WITHOUT TIME ZONE '0001-01-01 00:00:00.0Z');

("2015-01-01 00:00:00+00","0001-01-01 00:00:00+00","0001-01-01 00:00:00")

No Year Zero

By the way, the date portion seems to be correct. It seems there is no year 0000, that being the pivot point between the “BC” and “AD” eras. Take the first moment of year 0001, subtract an hour, and you get the year 0001 BC – so no year zero.

SET TIME ZONE 'UTC' ;

INSERT INTO moment_  -- TIMESTAMP WITH TIME ZONE.
VALUES ( TIMESTAMP '0001-01-01 00:00:00.0Z' - INTERVAL '1 hour' ) ;

SET TIME ZONE 'UTC' ;

TABLE moment_ ;

The result is the year 0001 BC, so we jump from 0001 to 0001 BC; no year zero 0000.

"0001-12-31 23:00:00+00 BC"

I want to return a single column value in PostgreSQL function from a PostgreSQL database. How can I achieve that?

In Postgresql, when I try to run ALTER TYPE x ADD VALUE y; on a custom datatype I get an error stating PG::Error: ERROR: must be owner of type x.

I know this can be solved by running ALTER TYPE x OWNER TO <database owner> but what I’m curious about is how I can check the current owner of this specific datatype. dT+ x does not give me information about the current owner.

How can I extract more information about custom data types, including the current owner?

My current database uat has as owner postgres. But now I would like to set roles and users with the correct permissions. Something simple like:

roles: dba, developer, application

users: dba1, dev1, dev2, app1

Where dba can do anything on the database for administration purposes; developer can makes SELECT, UPDATE and INSERT, usual developer operations; app1 the same operations as the developer role, for now.

How can I make this correctly ? And who should be the owner of my production table in this new setup ?

UPDATE:

------------------------
-- INITIAL SETUP
------------------------

-- create uat database
create database uat;

-- dba role
create role dba with superuser createdb createrole nologin replication bypassrls;

-- dev role
create role dev with nosuperuser nocreatedb nocreaterole nologin noreplication nobypassrls;

-- app role
create role app with nosuperuser nocreatedb nocreaterole nologin noreplication nobypassrls;

-- alter postgres password
alter role postgres encrypted password '';

-- create users

create user dev1 login inherit encrypted password '' in role dev;
create user dev2 login inherit encrypted password '' in role dev;
create user dev3 login inherit encrypted password '' in role dev;
create user dev4 login inherit encrypted password '' in role dev;

create user webapp encrypted password '' in role app;

-- grant privileges to dba role
grant all privileges on all tables in schema public to dba;

------------------------
-- RESTORE 
------------------------
psql -U postgres -d uat -h localhost -W -p 5432 < uat.sql 

But when I logged in as user dev1 for example, I can’t do anything:

select * from guru_basket_security;

Listing access for this table:

    z    
                                                              Access privileges
     Schema |                         Name                         |   Type   |     Access privileges     | Column privileges | Policies 
    --------+------------------------------------------------------+----------+---------------------------+-------------------+----------
     public | guru_basket_security                                 | table    |                           |                   | 
     public | identifier                                           | table    | postgres=arwdDxt/postgres |                   | 

I got these errors:

ERROR:  permission denied for relation guru_basket_security
ERROR:  permission denied for relation identifier

I am running Postgres 9.3 on Ubuntu 14.04 and I am unable to get pgAgent daemon/service to run.

I have confirmed that the daemon is not running using service --status-all.

When trying to run pgagent (whilst logged in as the only user in ubuntu) using pgagent hostaddr=127.0.0.1 dbname=postgres user=user1 the following messages are displayed:

WARNING: Couldn't create the primary connection (attempt 10): fe_sendauth: no password supplied

ERROR: Stopping pgAgent: Couldn't establish the primary connection with the database server.

The lack of running pgAgent was also confirmed when trying to run a job on demand.

The user ‘user1′ is able to log into pgAgmin and connect to the database from other computers on the local network.

The user looks like this in the pg_hba.conf file:

# Database administrative login by Unix domain socket
local   all             user1                                md5

A .pgpass password file also exists for this user in the home directory.

What is causing the above messages and what needs to be done to get pgAgent to run scheduled tasks?

I have these 2 tables created in postgres,

CREATE TABLE EMPLOYEE(
   ID           INT             PRIMARY KEY NOT NULL,
   NAME         VARCHAR(100)    NOT NULL,
   ADDRESS      VARCHAR (250)   NOT NULL
);


CREATE TABLE HIST_EMPLOYEE(
   HISTORYNUM   INT             NOT NULL,
   ID           INT             NOT NULL,
   NAME         VARCHAR(100)    NOT NULL,
   ADDRESS      VARCHAR (250)   NOT NULL
);

..so whenever I want to insert, update, or delete a record in my EMPLOYEE table, that record should be inserted in HIST_EMPLOYEE table. regardless if it is single or multiple rows

..I know that there are lots of answers in this site..but the problem is I added a column in my HISTORY table HIST_EMPLOYEE..that’s why I cant used this script that most of the answers posted,

insert into hist_employee select * from employee

…this is what i’ve started so far, but it has so many errors..

TRIGGER

CREATE TRIGGER insertHistory
    AFTER INSERT OR UPDATE ON employee   --MAIN TABLE
        EXECUTE PROCEDURE insertHistory('hist_employee');  --HISTORY TABLE

TRIGGER FUNCTION

CREATE OR REPLACE FUNCTION insertHistory() RETURNS TRIGGER AS 
$func$
    BEGIN
       EXECUTE format('insert into %I select * from %I', TG_ARGV[0] ,TG_TABLE_NAME );
       return null;
    END;
$func$ 
LANGUAGE plpgsql;

NOTE:

1. This trigger should be also applicable or flexible enough to be implemented on other tables with different set of columns.

2. HISTORYNUM column is unique per id,

3. And, is it possible to combine these 3, AFTER INSERT, AFTER UPDATE, BEFORE DELETE .. in one trigger ??

thanks to all who will respond..sorry if you find my question too vague

I’m using the postgres.app on my Mac Mini Server running version 9.4.0 using El Capitan. I have not done any customization in pg-hba.conf. I do not have a .pgpass file. I believe that .pgpass is used to store passwords where you would not have to enter them when accessing a database instance. I really need more detailed understanding on the Unix terminology being used in pg-hba.conf.

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

For my Ruby on Rails projects having roles with passwords with different types of access will be sufficient. From what I can tell everything I want would be considered local.

I submitted the two questions below in Stack Overflow last year and tried to implement them but to no avail. I ended up either locking myself out of PostgesSQL or with no passwords working. At that time I was using the built-in PostgreSQL instance on my Mac Mini Server.

http://stackoverflow.com/questions/22882280/postgresql-9-2-4-desire-to-change-the-localhost-server-password-in-mavericks

http://stackoverflow.com/questions/25167284/rails-4-desire-to-set-postgresql-password-access-database-in-rails

I want to add an encrypted password to my current role. I also want to add two more roles with encrypted passwords to use in my Ruby on Rails applications hosted on my Mac Mini server, one with read only access and the other with read/write access.

I assume I would execute the following commands in the PostgreSQL terminal or in pgAdmin.

CREATE USER readonly WITH ENCRYPTED PASSWORD 'readonlypasswordencryptedtext';
CREATE USER readwrite WITH ENCRYPTED PASSWORD 'readwritepasswordencryptedtext';
GRANT SELECT PRIVILEGES ON ALL TABLES IN public TO readonly, readwrite;
GRANT INSERT, UPDATE, DELETE PRIVILEGES ON TABLE table1, table2, table3, etc. TO readwrite;
ALTER ROLE myuserid WITH ENCRYPTED PASSWORD 'myuseridpasswordencryptedtext';

I would like help on how to change pg-hba.conf and in what order so I don’t lock out myuserid AND if there are additional things I need to consider.

I have multiple databases with tables. I don’t think my GRANT statements take that into consideration. Would I need to create a schema to include all my databases for the readonly user and the databases to update for the readwrite user?

Is it actually possible to somehow use Slony with Amazon’s RDS service? I’m aware that there’s no such thing as a host that you can log into to make Slony’s configurations, but perhaps it can be done remotely from a side server or something like that?

I’m looking to install QGIS on my Linux/Fedora 21 laptop.

QGIS ver. 2.8 seems to be the new/stable version (Feb. 2016)

What PostgreSQL version should I use with that?

My system is Windows 7 with PostgreSQL 9.4 and PostGIS 2.2.

I’m running shp2pgsql program from command line (cmd window, not PSQL interface).

shp2pgsql "D:file.shp" schema.table > file.sql

I’m getting the following error message (sorry this is in french but it says System error: the programme cannot start for libiconv-2.dll file is missing on the computer. Try to reinstall the program in order to fix the bug):

I have then completely reinstalled PostgreSQL 9.4 and PostGIS 2.2. The libiconv-2.dll is properly installed inside the bin directory (C:Program FilesPostgreSQL9.4bin). I don’t understand why this file is not found on the computer and I don’t know what else I could do…

So I am trying to do a where clause on a select and my code is

ST_Within(shape::GEOMETRY, ST_Buffer(ST_GeogFromText('@Value(_location)'), 5, 'endcap=round join=round')

I am trying to buffer the _location attribute by 5 meters so when the ST_Within is used it will help match better. This code is suppose to look for for input points and see if they match anything in the data set already. via spatial match. Just FYI both points for ST_within are points.

If anyone has any idea how i used ST_Bufffer wrong that would be great help.

I need a PostgreSQL query that do the following job: given a percentage (e.g. 10%), the query should return each 1st element of each percentile, among all the elements selected.

So suppose in this simple case my table contains elements which are integer values between 1 and 100:

id    | value
--    | -----
id1   | 1
id2   | 2
...
id100 | 100

And suppose the input percentile is 10%.
The query should divide all the dataset into spans whose range is 10%, and return each first element.
So the output should be:

id1 | 1
id11 | 11
id21 | 21
id31 | 31
id41 | 41
id51 | 51
id61 | 61
id71 | 71
id81 | 81
id91 | 91

Does anyone knows how I could write this Sql query? Thanks!

I have an application where several devices send data every second to my server. On the database I have a table with the devices and another table with the data sent every second.

The first time a device sends the data the server script should register (INSERT) that device id in the device table and add the data to the data table.

My question is, what would be faster?

1. To do a SELECT EXISTS query to determine whether an INSERT is required on the device table. If SELECT EXISTS returns false then execute an INSERT, otherwise do nothing.

2. To execute always an INSERT statement. Considering the device id is primary key on the table, if the device already exists it would return an error, otherwise it would insert it.

Only the first time a device sends data would require an actual INSERT, after that no INSERT would ever be required.

The PostgreSQL version is 9.4.

The table of devices is defined as follows:

CREATE TABLE common.tag
(
  customer integer,
  hostname character varying(150),
  description text,
  model integer,
  configprofile integer,
  id character varying(150) NOT NULL,
  host integer,
  CONSTRAINT tag_pk PRIMARY KEY (id),
  CONSTRAINT tag_fk_client FOREIGN KEY (customer)
      REFERENCES common.customer (id) MATCH SIMPLE
      ON UPDATE NO ACTION ON DELETE NO ACTION,
  CONSTRAINT tag_fk_configprofile FOREIGN KEY (configprofile)
      REFERENCES common.configprofile (id) MATCH SIMPLE
      ON UPDATE NO ACTION ON DELETE NO ACTION,
  CONSTRAINT tag_fk_tagmodel FOREIGN KEY (model)
      REFERENCES common.tagmodel (id) MATCH SIMPLE
      ON UPDATE NO ACTION ON DELETE NO ACTION
)

I’ve asked this question over in SE GIS, but told it would be better suited to this SE, so here goes…

I know Postgres is brilliant for grouping points based on their spatial values – that I can do – but is there a way I can group points by their ‘closeness’ in time. By this I mean, how would I go about identifying points that were created within x minutes of the first point.

To put it into some context, the first point could be the first Facebook status update immediately after Arsenal scored against Aston Villa. There would be a sharp rise in status updates immediately after the goal before gradually trailing off.

So..If I had a dataset of 50,000 records representing Facebook status updates each with a random time and then I edited 1000 of these to be within 10 minutes of the first post, how would I best approach this and what would the SQL look like?

The time format is arranged as YY:MM:DD HH:MM:SS.

So far I have been able to return those records which are close spatially and have a similar message, but adding in the time element so I have results that are close spatially, temporally and share the same topic has proved a little more difficult for me.

I’ve looked at the interval function but I am unsure of how exactly to deploy this in my SQL. Would it be that first I filter records by their spatial dispersal, then filter which are close in time and finally the similarity of their status?

My SQL looks something like this for now..

Creating the table…

CREATE TABLE FB_status (
id character varying (254),
status character varying (254),
longitude numeric,
latitude numeric,
posted_at character varying (254),
geom geometry(POINT));

My SQL thus far…

SELECT (a.status, b.status) 
FROM FB_status AS a 
JOIN FB_status AS b 
ON ST_Dwithin (a.geom, b.geom, 0.01)
WHERE similarity (a.status, b.status) > 0.5;

I have the follwing situation: I have profiles and tasks. One profile can be used once for each task. I track usage in profiles_used table. I need to build a report that shows how many profiles of each group is available for each task.

Here are my tables:

select * from profiles;

 id |  name   | group
----+---------+---------
  1 |  name_1 | 1
  2 |  name_2 | 2
  3 |  name_3 | 3
  4 |  name_4 | 3
  5 |  name_5 | 3


select * from tasks;

 id | name
----+--------
  1 | task_1
  2 | task_2
  3 | task_3


select * from profiles_used;

 id | task.id | profile.id
----+---------+------------
  0 | 1       | 1
  1 | 1       | 2
  2 | 2       | 1
  3 | 3       | 4
  3 | 3       | 5

Magic: Run query that finds out how many not used profiles in each group available for each task.

Result:

             profiles.group
 task.id |  1  |  2  |  3  
---------+-----+-----+-----
       1 |  0  |  0  |  3
       2 |  0  |  1  |  3
       3 |  1  |  1  |  1

Please help me with right direction to start.

How would I write an ALTER statement or otherwise to convert an existing imported data column from ” wkb_geometry geometry(LineString,4326)” to a geom data type?

I ask because I’ve moved to importing my data from geojson instead of shapefiles due to column length issues with shapefiles and now I’ve created the new problem for myself of not the data type not being geom as default.

So I’ve been doing a stack of reading. I’m still not 100% clear of the best way to go about it.

Let’s take a simple setup.
3 nodes to server UWSGI apps (in random), 3 nodes for PGSQL, 1 nominal master and 2 read only slaves – it’s not setup yet but that’s the goal. 80% of the requests are read only anyway for most apps.

Now what really bugs me is that PGPool is usually illustrated as a single machine. If we are going for high availability as the primary goal with load balancing as a highly desirable side effect, introducing the need for a PGPool in the middle is introducing a singular point of failure as well as perhaps an otherwise unecesaey node.

Trying to ascertain what to do I imagine that really the app servers could be PGpools as they’re the first one to notice if a DB goes down, but I’ve heard multiple PGPools are
Complicated.

I mean what I really want is the built in replication ability with the ability to execute a failover and mark the new master etc, I know PGPool does it. I’d like it to be decentralised if possible – can people please point me in the right direction.?

My data looks like this:

queryable_type   queryable_id 
User             1 
User             2
User             2

I’d like to return dupes so that my result would look like this:

queryable_type  queryable_id count 
User            2            2

I’m assuming some multiple group by but it wasn’t working.