You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

445 lines
26 KiB

This file contains invisible Unicode characters!

This file contains invisible Unicode characters that may be processed differently from what appears below. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to reveal hidden characters.

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

## 一、常用命令及SQL
+ 系统性能指标
```sql
1.当前连接数
select count(*) from pg_stat_activity where query <>'COMMIT';
2.查看所有表
select * from pg_tables where schemaname='public';
3.查看表字段
select * from information_schema.columns where table_schema='public' and table_name='device';
4.查看表大小(包括索引)
select pg_size_pretty(pg_total_relation_size('device'))
5.使用表扫描最多的表
select * from pg_stat_user_tables where n_live_tup > 100000 and seq_scan > 0 order by seq_tup_read desc limit 10;
6.查看所有表大小
select relname, pg_size_pretty(pg_relation_size(relid)) as size from pg_stat_user_tables;
7.查看死锁
select * from pg_locks where granted='f'
8.结束死锁进程
select pg_terminate_backend(pid) from (select pid from pg_locks where granted='f')
9.查看表结构
SELECT
a.attname as column_name,
format_type(a.atttypid,a.atttypmod) as data_type,
col_description(a.attrelid,a.attnum) as comment
FROM pg_class as c,pg_attribute as a where a.attrelid = c.oid and a.attnum>0 and c.relname = 'Edge_0101';
-- 或者
SELECT a.attnum, a.attname AS field, t.typname AS type, a.attlen AS length, a.atttypmod AS lengthvar
, a.attnotnull AS notnull, b.description AS comment
FROM pg_class c, pg_attribute a
LEFT JOIN pg_description b
ON a.attrelid = b.objoid
AND a.attnum = b.objsubid, pg_type t
WHERE c.relname = 'pg_replication_slots'
AND a.attnum > 0
AND a.attrelid = c.oid
AND a.atttypid = t.oid
ORDER BY a.attnum;
10.查看所有进程
SELECT * FROM pg_stat_activity WHERE datname='数据库名'
11.根据pid结束进程
select pg_cancel_backend(pid)
```
## 二、函数
### 1.字符串
+ 字符串拼接
```sql
'Post' || 'greSQL'  PostgreSQL
SELECT CONCAT ('CONCAT',' ', 'function')
```
+ 转大、小写
```sql
select upper('tom')
select lower('ABC')
```
+ 子串在字符串中的位置,可以替代like
```sql
select position('om' in 'Thomas')
```
+ 字符串截取
```sql
# 从下标0开始截取4个字符
select substr('month',0,4)
```
+ 去空格(正则)
```sql
select regexp_replace('杂 项 制 品', E'\\s+', '', 'g')
```
### 2.日期时间
+ 获取当前时间绝对秒
```sql
select floor(extract(epoch from now()))
```
+ 日期转字符串
```sql
select to_char(now(),'yyyy-MM-dd HH24:mi:ss')
```
+ 字符串转日期
```sql
select to_date('1992-04-12','yyyy-MM-dd')
```
+ 当前时间、时间戳
```sql
select current_timestamp,now();
```
+ 字符串转时间戳
```sql
select to_timestamp('2024-03-21 14:03:14','yyyy-MM-dd HH24:mi:ss')
```
+ 获取当前日期
```sql
select current_date;
```
+ 获取6个月之前时间
```sql
select to_char((select now() - interval '6 month'),'yyyy-MM-dd hh24:mi:ss')
```
+ 计算时间差
```sql
1.天数(day可以换成month,year
select date_part('day','2019-01-01'::date-'2011-10-02'::date)
2.
(extract('epoch' from a.pass_time::timestamp) - extract('epoch' from b.snap_time::timestamp))
```
### 3.窗口函数
窗口函数不是将一组数据汇总为单个结果;而是针对每一行数据,基于和它相关的一组数据计算出一个结果
![[Pasted image 20240501095744.png]]
窗口函数的定义如下:
```text
window_function ( expression, ... ) OVER (
PARTITION BY ...
ORDER BY ...
frame_clause
)
```
| 函数类型 | 名称 | 用途 |
| ---- | ------------ | ---------------------------------------------------- |
| 聚合函数 | AVG | 平均值 |
| 聚合函数 | COUNT | 行数 |
| 聚合函数 | MAX | 最大值 |
| 聚合函数 | MIN | 最小值 |
| 聚合函数 | SUM | 总数 |
| 排名函数 | ROW_NUMBER | 为分区中的每行数据分配一个序列号,序列号从 1 开始分配 |
| 排名函数 | RANK | 计算每行数据在其分区中的名次;如果存在名次相同的数据,后续的排名将会产生跳跃 |
| 排名函数 | DENSE_RANK | 计算每行数据在其分区中的名次;即使存在名次相同的数据,后续的排名也是连续的值 |
| 排名函数 | PERCENT_RANK | 以百分比的形式显示每行数据在其分区中的名次;如果存在名次相同的数据,后续的排名将会产生跳跃 |
| 排名函数 | CUME_DIST | 计算每行数据在其分区内的累积分布,也就是该行数据及其之前的数据的比率;取值范围大于 0 并且小于等于 1 |
| 排名函数 | NTILE | 将分区内的数据分为 N 等份,为每行数据计算其所在的位置 |
| 取值函数 | FIRST_VALUE | 返回窗口内第一行的数据 |
| 取值函数 | LAST_VALUE | 返回窗口内最后一行的数据 |
| 取值函数 | NTH_VALUE | 返回窗口内第 N 行的数据 |
| 取值函数 | LAG | 返回分区中当前行之前的第 N 行的数据 |
| 取值函数 | LEAD | 返回分区中当前行之后第 N 行的数据 |
#### 3.1 排名窗口函数
以下示例按照部门为单位,计算员工的月薪排名:
```sql
SELECT d.department_name "部门名称", concat(e.first_name, ',' , e.last_name) "姓名", e.salary "月薪",
ROW_NUMBER() OVER (PARTITION BY e.department_id ORDER BY e.salary DESC) AS "row_number",
RANK() OVER (PARTITION BY e.department_id ORDER BY e.salary DESC) AS "rank",
DENSE_RANK() OVER (PARTITION BY e.department_id ORDER BY e.salary DESC) AS "dense_rank",
PERCENT_RANK() OVER (PARTITION BY e.department_id ORDER BY e.salary DESC) AS "percent_rank"
FROM employees e
JOIN departments d ON (e.department_id = d.department_id)
WHERE d.department_name in ('IT', 'Purchasing')
ORDER BY 1, 4;
```
![[Pasted image 20240501102554.png]]
ROW_NUMBER 函数为每个员工分配了一个连续的数字编号可以看作是一种排名。IT 部门的“Valli,Pataballa”和“David,Austin”的月薪相同但是编号不同
RANK 函数为每个员工指定了一个名次IT 部门的“Valli,Pataballa”和“David,Austin”的名次都是 3而且在他们之后的“Diana,Lorentz”的名次为 5产生了跳跃
DENSE_RANK 函数为每个员工指定了一个名次IT 部门的“Valli,Pataballa”和“David,Austin”的名次都是 3在他们之后的“Diana,Lorentz”的名次为 4名次是连续值
PERCENT_RANK 函数按照百分比指定名次,取值位于 0 到 1 之间。其中“Diana,Lorentz”的百分比排名为 1也产生了跳跃。
以上SQL的简化写法:
```sql
SELECT d.department_name "部门名称", concat(e.first_name, ',' , e.last_name) "姓名", e.salary "月薪",
ROW_NUMBER() OVER w AS "row_number",
RANK() OVER w AS "rank",
DENSE_RANK() w AS "dense_rank",
PERCENT_RANK() w AS "percent_rank"
FROM employees e
JOIN departments d ON (e.department_id = d.department_id)
WHERE d.department_name in ('IT', 'Purchasing')
WINDOW w AS (PARTITION BY e.department_id ORDER BY e.salary DESC)
ORDER BY 1, 4;
```
其中,`WINDOW`定义了一个窗口变量 w然后在窗口函数的 OVER 子句中使用了该变量;这样可以简化函数的输入。
以下语句演示了 CUME_DIST 和 NTILE 函数的作用:
```sql
SELECT concat(first_name, ',' , last_name) "姓名", hire_date AS "入职日期",
CUME_DIST() OVER (ORDER BY hire_date) AS "累积占比",
NTILE(100) OVER (ORDER BY hire_date) AS "相对位置"
FROM employees;
```
![[Pasted image 20240501102825.png]]
其中CUME_DIST 函数显示 2001-01-13 以及之前入职的员工大概有 0.9%1/107NTILE(100) 函数表明前 1% 入职的员工有“Lex,De Haan”和“Hermann,Baer”由于员工总数为 107所以不是完全准确。
#### 3.2 取值窗口函数
以下语句使用 FIRST_VALUE、LAST_VALUE 以及 NTH 函数分别获取每个部门内部月薪最高、月薪最低以及月薪第三高的员工:
```sql
SELECT department_id, first_name, last_name, salary,
FIRST_VALUE(salary) OVER w,
LAST_VALUE(salary) OVER w,
NTH_VALUE(salary, 3) OVER w
FROM employees
WINDOW w AS (PARTITION BY department_id ORDER BY salary desc ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
ORDER BY department_id, salary DESC;
```
![[Pasted image 20240501104752.png]]
LAG 和 LEAD 函数同样用于计算销量数据的环比/同比增长
```sql
WITH sales_monthly AS (
SELECT product, to_char(saledate,'YYYYMM') ym, sum(amount) sum_amount
FROM sales_data
GROUP BY product, to_char(saledate,'YYYYMM')
)
SELECT product AS "产品", ym "年月", sum_amount "销量",
(sum_amount - LAG(sum_amount, 1) OVER (PARTITION BY product ORDER BY ym))/
LAG(sum_amount, 1) OVER (PARTITION BY product ORDER BY ym) * 100 AS "环比增长率(%"
FROM sales_monthly
ORDER BY product, ym;
```
![[Pasted image 20240501105012.png]]
首先,创建一个通用表表达式 sales_monthly得到了不同产品每个月的销量汇总LAG(sum_amount, 1) 表示获取上一期的销量;当前月份的销量减去上个月的销量,再除以上个月的销量,就是环比增长率。
### 4.正则表达式
```sql
-- 提取数字
NULLIF(regexp_replace(po_number, '\D','','g'), '')::numeric
```
## 三、索引
[PostgreSQL 14种索引的原理和应用场景 - 简书 (jianshu.com)](https://www.jianshu.com/p/6aedd1b79dba)
| 索引 | 原理 | 场景 | 适用的数据类型 | 版本限制 |
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ---- |
| btree | b-tree适合所有的数据类型支持排序支持大于、小于、等于、大于或等于、小于或等于的搜索 | | 所有 | 无 |
| hash | hash索引存储的是被索引字段VALUE的哈希值只支持等值查询。 | | 所有 | 无 |
| gin | 当需要搜索多值类型内的VALUE时适合多值类型例如数组、全文检索、TOKEN。根据不同的类型支持相交、包含、大于、在左边、在右边等搜索 | | | |
| gist | GiST是一个通用的索引接口可以使用GiST实现b-tree, r-tree等索引结构。不同的类型支持的索引检索也各不一样。 | 例如: <br>1、几何类型支持位置搜索包含、相交、在上下左右等按距离排序。 <br>2、范围类型支持位置搜索包含、相交、在左右等<br>3、IP类型支持位置搜索包含、相交、在左右等<br>4、空间类型PostGIS支持位置搜索包含、相交、在上下左右等按距离排序。 <br>5、标量类型支持按距离排序 | | |
| sp-gist | SP-GiST类似GiST是一个通用的索引接口但是SP-GIST使用了空间分区的方法使得SP-GiST可以更好的支持非平衡数据结构例如quad-trees, k-d tree, radis tree。 | 应用场景 <br>1、几何类型支持位置搜索包含、相交、在上下左右等按距离排序。 <br>2、范围类型支持位置搜索包含、相交、在左右等<br>3、IP类型支持位置搜索包含、相交、在左右等。 | | |
| brin | BRIN 索引是块级索引有别于B-TREE等索引BRIN记录并不是以行号为单位记录索引明细而是记录每个数据块或者每段连续的数据块的统计信息。因此BRIN索引空间占用特别的小对数据写入、更新、删除的影响也很小。 | 时序数据在时间或序列字段创建BRIN索引进行等值、范围查询时效果很棒。 | | |
| rum | rum 是一个索引插件由Postgrespro开源适合全文检索属于GIN的增强版本。 <br>增强包括: <br>1、在RUM索引中存储了lexem的位置信息所以在计算ranking时不需要回表查询而GIN需要回表查询<br>2、RUM支持phrase搜索而GIN无法支持。 <br>3、在一个RUM索引中允许用户在posting tree中存储除ctid行号以外的字段VALUE例如时间戳。 | RUM不仅支持GIN支持的全文检索还支持计算文本的相似度值按相似度排序等。同时支持位置匹配例如速度与激情可以采用"速度" <2> "激情" 进行匹配而GIN索引则无法做到 | | |
| bloom | bloom索引接口是PostgreSQL基于bloom filter构造的一个索引接口属于lossy索引可以收敛结果集(排除绝对不满足条件的结果,剩余的结果里再挑选满足条件的结果)因此需要二次checkbloom支持任意列组合的等值查询。 <br>bloom存储的是签名签名越大耗费的空间越多但是排除更加精准。有利有弊 | | | |
| zombodb | zombodb是PostgreSQL与ElasticSearch结合的一个索引接口可以直接读写ES | 与ES结合实现SQL接口的搜索引擎实现数据的透明搜索 | | |
| bitmap | bitmap索引是Greenplum的索引接口类似GIN倒排只是bitmap的KEY是列的值VALUE是BIT每个BIT对应一行而不是行号list或tree | 当某个字段的唯一值个数在100到10万之间(超出这个范围不建议使用bitmap)时如果表的记录数特别多而且变更不频繁或者是AO表那么很适合BITMAP索引bitmap索引可以实现快速的多个或单个VALUE的搜索。因为只需要对行号的BITMAP进行BIT与或运算得到最终的BITMAP从最终的BITMAP映射到行进行提取。 <br>bitmap与btree一样都支持 等于,大于,小于,大于等于,小于等于的查询。 | | |
| 表达式索引 | 表达式索引也是PostgreSQL特有的特性例如用户的数据需要转换后查询例如某些设备上传的地理坐标的坐标系不符合国标需要转换为国内的空间坐标来查询。 <br>那么可以针对这类字段,创建表达式索引,将转换过程放到表达式中,查询时也使用表达式进行查询。 | CREATE INDEX ods_ht_transport_realtime_pass_time_idx ON ods_ht_transport_realtime ((pass_time::DATE)); 函数索引 | | |
## 四、事务
## 五、常见问题
## 六、常用函数
+ 生成建表语句
```sql
CREATE OR REPLACE FUNCTION show_create_table(
in_schema_name varchar,
in_table_name varchar
)
RETURNS text
LANGUAGE plpgsql VOLATILE
AS
$$
DECLARE
-- the ddl we're building
v_table_ddl text;
-- data about the target table
v_table_oid int;
v_table_type char;
v_partition_key varchar;
v_table_comment varchar;
-- records for looping
v_column_record record;
v_constraint_record record;
v_index_record record;
v_column_comment_record record;
v_index_comment_record record;
v_constraint_comment_record record;
BEGIN
-- grab the oid of the table; https://www.postgresql.org/docs/8.3/catalog-pg-class.html
SELECT c.oid, c.relkind INTO v_table_oid, v_table_type
FROM pg_catalog.pg_class c
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind in ('r', 'p')
AND c.relname = in_table_name -- the table name
AND n.nspname = in_schema_name; -- the schema
-- throw an error if table was not found IF (v_table_oid IS NULL) THEN
RAISE EXCEPTION 'table does not exist';
END IF;
-- start the create definition
v_table_ddl := 'CREATE TABLE "' || in_table_name || '" (' || E'\n';
-- define all of the columns in the table; https://stackoverflow.com/a/8153081/3068233
FOR v_column_record IN
SELECT c.column_name,
c.data_type,
c.character_maximum_length,
c.is_nullable,
c.column_default
FROM information_schema.columns c
WHERE (table_schema, table_name) = (in_schema_name, in_table_name)
ORDER BY ordinal_position
LOOP
v_table_ddl := v_table_ddl || ' ' -- note: two char spacer to start, to indent the column
|| '"' || v_column_record.column_name || '" '
|| v_column_record.data_type || CASE WHEN v_column_record.character_maximum_length IS NOT NULL THEN ('(' || v_column_record.character_maximum_length || ')') ELSE '' END || ' '
|| CASE WHEN v_column_record.is_nullable = 'NO' THEN 'NOT NULL' ELSE 'NULL' END
|| CASE WHEN v_column_record.column_default IS NOT null THEN (' DEFAULT ' || v_column_record.column_default) ELSE '' END
|| ',' || E'\n';
END LOOP;
-- define all the constraints in the; https://www.postgresql.org/docs/9.1/catalog-pg-constraint.html && https://dba.stackexchange.com/a/214877/75296
FOR v_constraint_record IN
SELECT con.conname as constraint_name,
con.contype as constraint_type,
CASE
WHEN con.contype = 'p' THEN 1 -- primary key constraint
WHEN con.contype = 'u' THEN 2 -- unique constraint
WHEN con.contype = 'f' THEN 3 -- foreign key constraint
WHEN con.contype = 'c' THEN 4
ELSE 5
END as type_rank,
pg_get_constraintdef(con.oid) as constraint_definition
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class rel ON rel.oid = con.conrelid
JOIN pg_catalog.pg_namespace nsp ON nsp.oid = connamespace
WHERE nsp.nspname = in_schema_name
AND rel.relname = in_table_name
ORDER BY type_rank
LOOP
IF v_constraint_record.constraint_type = 'p' THEN
v_table_ddl := v_table_ddl || ' '
|| v_constraint_record.constraint_definition
|| ',' || E'\n';
ELSE
v_table_ddl := v_table_ddl || ' ' -- note: two char spacer to start, to indent the column
|| 'CONSTRAINT' || ' '
|| '"' || v_constraint_record.constraint_name || '" '
|| v_constraint_record.constraint_definition
|| ',' || E'\n';
END IF;
END LOOP;
-- drop the last comma before ending the create statement
v_table_ddl = substr(v_table_ddl, 0, length(v_table_ddl) - 1) || E'\n';
-- end the create definition
v_table_ddl := v_table_ddl || ')';
IF v_table_type = 'p' THEN
SELECT pg_get_partkeydef(v_table_oid) INTO v_partition_key;
IF v_partition_key IS NOT NULL THEN
v_table_ddl := v_table_ddl || ' PARTITION BY ' || v_partition_key;
END IF;
END IF;
v_table_ddl := v_table_ddl || ';' || E'\n';
-- suffix create statement with all of the indexes on the table
FOR v_index_record IN
SELECT regexp_replace(indexdef, ' "?' || schemaname || '"?\.', ' ') AS indexdef
FROM pg_catalog.pg_indexes
WHERE (schemaname, tablename) = (in_schema_name, in_table_name)
AND indexname NOT IN (
select con.conname
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class rel ON rel.oid = con.conrelid
JOIN pg_catalog.pg_namespace nsp ON nsp.oid = connamespace
WHERE nsp.nspname = in_schema_name
AND rel.relname = in_table_name
)
LOOP
v_table_ddl := v_table_ddl
|| v_index_record.indexdef
|| ';' || E'\n';
END LOOP;
-- comment on table
SELECT description INTO v_table_comment
FROM pg_catalog.pg_description
WHERE objoid = v_table_oid AND objsubid = 0;
IF v_table_comment IS NOT NULL THEN
v_table_ddl := v_table_ddl || 'COMMENT ON TABLE "' || in_table_name || '" IS ''' || replace(v_table_comment, '''', '''''') || ''';' || E'\n';
END IF;
-- comment on column
FOR v_column_comment_record IN
SELECT col.column_name, d.description
FROM information_schema.columns col
JOIN pg_catalog.pg_class c ON c.relname = col.table_name
JOIN pg_catalog.pg_namespace nsp ON nsp.oid = c.relnamespace AND col.table_schema = nsp.nspname
JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = col.ordinal_position
WHERE c.oid = v_table_oid
ORDER BY col.ordinal_position
LOOP
v_table_ddl := v_table_ddl || 'COMMENT ON COLUMN "' || in_table_name || '"."'
|| v_column_comment_record.column_name || '" IS '''
|| replace(v_column_comment_record.description, '''', '''''') || ''';' || E'\n';
END LOOP;
-- comment on index
FOR v_index_comment_record IN
SELECT c.relname, d.description
FROM pg_catalog.pg_index idx
JOIN pg_catalog.pg_class c ON idx.indexrelid = c.oid
JOIN pg_catalog.pg_description d ON idx.indexrelid = d.objoid
WHERE idx.indrelid = v_table_oid
LOOP
v_table_ddl := v_table_ddl || 'COMMENT ON INDEX "'
|| v_index_comment_record.relname || '" IS '''
|| replace(v_index_comment_record.description, '''', '''''') || ''';' || E'\n';
END LOOP;
-- comment on constraint
FOR v_constraint_comment_record IN
SELECT con.conname,
pg_description.description
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class rel ON rel.oid = con.conrelid
JOIN pg_catalog.pg_namespace nsp ON nsp.oid = connamespace
JOIN pg_catalog.pg_description ON pg_description.objoid = con.oid
WHERE rel.oid = v_table_oid
LOOP
v_table_ddl := v_table_ddl || 'COMMENT ON CONSTRAINT "'
|| v_constraint_comment_record.conname || '" ON "' || in_table_name || '" IS '''
|| replace(v_constraint_comment_record.description, '''', '''''') || ''';' || E'\n';
END LOOP;
-- return the ddl
RETURN v_table_ddl;
END
$$;
```
```sql
-- 需要指定schema
select show_create_table('public', 'device')
```