Oracle 12c Extended Datatypes

Extended Datatypes

Prior to Oracle 12c, VARCHAR2/NVARCHAR2 datatypes  allowed storing variable length characters of up to 4000 bytes whereas RAW data type  allowed storing variable length binry data of up to  2000 bytes.  In Oracle 12c, the maximum size for data types VARCHAR2, NVARCHAR2  and RAW is increased to 32767 bytes.  These data types are now referred to as extended data types. Extended data types are implicitly implemented using concept of LOBs;  Just like LOB’s, any data stored in excess of  4000 bytes for VARCHAR2/NVARCHAR2/RAW is stored out-line. If the data stored is within pre-Oracle12c limits , then they are stored in-line.  So be aware about the restrictions of LOBs since all most all of the restrictions of lobs are applicable to  extended datatypes (at least for now).

The first thing that came to mind about extended data types is what purpose does it actually serve since it is implemented as LOB; Why would you introduce LOB restrictions to your columns by increasing the limits of VARCHAR2, NVARCHAR2  and RAW. As most of you are aware that all versions of databases including Oracle 12c do  not support changing VARCHAR2/NUMBER/RAW  to LOBs directly with ALTER ..MODIFY command. (Would result in ORA-22858: invalid alteration of datatype); I thought that  instead of  extended datatype ,Oracle could have removed the restriction to convert VARCHAR2/NUMBER/RAW  to LOBs directly and provided some restricted form of index support.

Being a new feature, I did not find much useful information about the feature and so decided why not be one of the earliest blogger to explore this topic.

So let us make sure that  the database is  enabled for extended datatypes.

SQL> show parameter max_string_size

NAME                                 TYPE        VALUE

———————————— ———– ——————————

max_string_size                      string      EXTENDED

Please refer to following blog on instructions of how to set up your database for extended datatypes. https://twelvec.com/2014/02/08/oracle-12c-how-to-setup-your-database-to-support-extended-datatypes/

Now lets create a table with two columns , one with VARCHAR2 and other with LOB

SQL> CONN SHAN/SHAN

Connected.

SQL> CREATE TABLE EXT_DATATYPE (

2  EMPLOYEE_NO VARCHAR2(32767) ,

3  EMPLOYEE_COMMENTS  BLOB);

Table created.

 Ok now lets trying adding primary key to this table. Ooops , It failed because of the restrictions on  index lengths.

SQL> ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_NO);

ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_NO)

*

ERROR at line 1:

ORA-01450: maximum key length (6398) exceeded

Now lets modify the column size to 6398 bytes and add primary key to the table. The command still fails; So what is the secret column length that supports indexes? It is actually  6389 bytes.

SQL>  ALTER TABLE EXT_DATATYPE MODIFY  (EMPLOYEE_NO VARCHAR2(6398));

Table altered.

SQL>  ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_NO);

ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_NO)

ERROR at line 1:

ORA-01450: maximum key length (6398) exceeded

For VARCHAR2/NVARCHR2/RAW columns more that 6389 bytes , you can create indexes using function based indexes or virtual columns. With both approaches, you are shortening the length of column to create indexes using either SUBSTR or new STANDARD_HASH functions or something creative that you can come up with.

Let us revert  the column length to  time trusted 4000 bytes, add index and then try to increase the column length. You are going to get a different error but the underlying cause us same.

SQL>  ALTER TABLE EXT_DATATYPE MODIFY  (EMPLOYEE_NO VARCHAR2(4000));

Table altered.

SQL>  ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_NO);

Table altered.

SQL> ALTER TABLE EXT_DATATYPE MODIFY  (EMPLOYEE_NO VARCHAR2(6390));

ALTER TABLE EXT_DATATYPE MODIFY  (EMPLOYEE_NO VARCHAR2(6390))

*

ERROR at line 1:

ORA-01404: ALTER COLUMN will make an index too large

Now let us drop our primary key constraint and  try adding lob data type as primary key; It will fail as LOB columns cannot be primary key or unique key.

SQL> alter table EXT_DATATYPE drop primary key;

Table altered.

SQL> ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_COMMENTS));

ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_COMMENTS))

ERROR at line 1:

ORA-01735: invalid ALTER TABLE option

SQL> ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_COMMENTS);

ALTER TABLE EXT_DATATYPE  ADD PRIMARY KEY (EMPLOYEE_COMMENTS)

ERROR at line 1:

ORA-02329: column of datatype LOB cannot be unique or a primary key

Summary: In essence, It if would have been better if Oracle increased the datatypes sizes to 6000 bytes instead of 32K so that indexing , primary key constraints are all supported.  There is always a learning curve with new features;  it gets little more complicated with enhancements.  With 32K sizes ,Initially it is going to introduce new risks and create confusion if implemented without understanding the actual consequences even though there are some workarounds for creating indexes using  virtual columns or function based indexes.  I just picked one restriction of LOBs ; More testing to be done to verify the behavior of AFTER UPDATE DML trigger , INSERT AS SELECT and many more restrictions.

Oracle 12c: How to setup your database to support extended datatypes?

This blog discusses the setup required to support extended datatypes in Oracle12c; For more information about extended datatypes, please refer to https://twelvec.com/2014/02/08/oracle12c_extended_datatypes/

In  nutshell , the following steps are required to enable extended datatypes. Most of the  steps are familiar except for step 3 and 4. In step 3 , we are modifying initialization parameter MAX_STRING_SIZE from STANDARD to EXTENDED. Once changed , this is a irreversible action.  Step 4  increases the sizes of VARCHAR2, NVARCHAR2 & RAW in the required views.

1. SHUTDOWN IMMEDIATE
2. STARTUP UPGRADE;
3. ALTER SYSTEM SET max_string_size=extended scope=spfile;
4. @?/rdbms/admin/utl32k.sql
5. SHUTDOWN IMMEDIATE;
6. STARTUP;

Example:

SQL> select name from v$database;

NAME

———

ORCL12C

SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL>

SQL>

SQL>

SQL> startup upgrade

ORACLE instance started.

Total System Global Area 2505338880 bytes

Fixed Size                  2291472 bytes

Variable Size             671090928 bytes

Database Buffers         1811939328 bytes

Redo Buffers               20017152 bytes

Database mounted.

Database opened.

SQL>

SQL>

SQL> ALTER SYSTEM SET max_string_size=extended;

System altered.

SQL> @?/rdbms/admin/utl32k.sql

Session altered.

DOC>#######################################################################

DOC>#######################################################################

DOC>   The following statement will cause an “ORA-01722: invalid number”

DOC>   error if the database has not been opened for UPGRADE.

DOC>

DOC>   Perform a “SHUTDOWN ABORT”  and

DOC>   restart using UPGRADE.

DOC>#######################################################################

DOC>#######################################################################

DOC>#

no rows selected

DOC>#######################################################################

DOC>#######################################################################

DOC>   The following statement will cause an “ORA-01722: invalid number”

DOC>   error if the database does not have compatible >= 12.0.0

DOC>

DOC>   Set compatible >= 12.0.0 and retry.

DOC>#######################################################################

DOC>#######################################################################

DOC>#

PL/SQL procedure successfully completed.

Session altered.

0 rows updated.

Commit complete.

System altered.

PL/SQL procedure successfully completed.

Commit complete.

System altered.

Session altered.

PL/SQL procedure successfully completed.

No errors.

Session altered.

PL/SQL procedure successfully completed.

Commit complete.

Package altered.

TIMESTAMP

——————————————————————————–

COMP_TIMESTAMP UTLRP_BGN  2014-01-28 14:51:31

DOC>   The following PL/SQL block invokes UTL_RECOMP to recompile invalid

DOC>   objects in the database. Recompilation time is proportional to the

DOC>   number of invalid objects in the database, so this command may take

DOC>   a long time to execute on a database with a large number of invalid

DOC>   objects.

DOC>

DOC>   Use the following queries to track recompilation progress:

DOC>

DOC>   1. Query returning the number of invalid objects remaining. This

DOC>      number should decrease with time.

DOC>         SELECT COUNT(*) FROM obj$ WHERE status IN (4, 5, 6);

DOC>

DOC>   2. Query returning the number of objects compiled so far. This number

DOC>      should increase with time.

DOC>         SELECT COUNT(*) FROM UTL_RECOMP_COMPILED;

DOC>

DOC>   This script automatically chooses serial or parallel recompilation

DOC>   based on the number of CPUs available (parameter cpu_count) multiplied

DOC>   by the number of threads per CPU (parameter parallel_threads_per_cpu).

DOC>   On RAC, this number is added across all RAC nodes.

DOC>

DOC>   UTL_RECOMP uses DBMS_SCHEDULER to create jobs for parallel

DOC>   recompilation. Jobs are created without instance affinity so that they

DOC>   can migrate across RAC nodes. Use the following queries to verify

DOC>   whether UTL_RECOMP jobs are being created and run correctly:

DOC>

DOC>   1. Query showing jobs created by UTL_RECOMP

DOC>         SELECT job_name FROM dba_scheduler_jobs

DOC>            WHERE job_name like ‘UTL_RECOMP_SLAVE_%’;

DOC>

DOC>   2. Query showing UTL_RECOMP jobs that are running

DOC>         SELECT job_name FROM dba_scheduler_running_jobs

DOC>            WHERE job_name like ‘UTL_RECOMP_SLAVE_%’;

DOC>#

PL/SQL procedure successfully completed.

TIMESTAMP

——————————————————————————–

COMP_TIMESTAMP UTLRP_END  2014-01-28 14:51:33

DOC> The following query reports the number of objects that have compiled

DOC> with errors.

DOC>

DOC> If the number is higher than expected, please examine the error

DOC> messages reported with each object (using SHOW ERRORS) to see if they

DOC> point to system misconfiguration or resource constraints that must be

DOC> fixed before attempting to recompile these objects.

DOC>#

OBJECTS WITH ERRORS

——————-

0

DOC> The following query reports the number of errors caught during

DOC> recompilation. If this number is non-zero, please query the error

DOC> messages in the table UTL_RECOMP_ERRORS to see if any of these errors

DOC> are due to misconfiguration or resource constraints that must be

DOC> fixed before objects can compile successfully.

DOC>#

ERRORS DURING RECOMPILATION

—————————

0

Function created.

PL/SQL procedure successfully completed.

Function dropped.

…Database user “SYS”, database schema “APEX_040200”, user# “98” 14:51:48

…Compiled 0 out of 2998 objects considered, 0 failed compilation 14:51:48

…263 packages

…255 package bodies

…453 tables

…11 functions

…16 procedures

…3 sequences

…458 triggers

…1322 indexes

…207 views

…0 libraries

…6 types

…0 type bodies

…0 operators

…0 index types

…Begin key object existence check 14:51:48

…Completed key object existence check 14:51:48

…Setting DBMS Registry 14:51:48

…Setting DBMS Registry Complete 14:51:48

…Exiting validate 14:51:48

PL/SQL procedure successfully completed.

SQL> SHUTDOWN IMMEDIATE

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL>

SQL>

SQL>

SQL> startup

ORACLE instance started.

Total System Global Area 2505338880 bytes

Fixed Size                  2291472 bytes

Variable Size             671090928 bytes

Database Buffers         1811939328 bytes

Redo Buffers               20017152 bytes

Database mounted.

Database opened.

SQL> show parameter max_string_size

NAME                                 TYPE        VALUE

———————————— ———– ——————————

max_string_size                      string      EXTENDED

PURGE DBA_RECYCLEBIN System Privilege in Oracle 12c

PURGE DBA_RECYCLEBIN  command empties the recyclebin at  database level i.e. purges the recyclebin for all users in the database.  Until 11g , you need to have powerful  SYSDBA system privilege to execute this command.  To some extent it made sense as this option was mainly used for upgrades or migration or to apply DST patch. One of focus areas of Oracle12c is separation of duties resulting in  new roles like SYSKM for wallet management, SYSBACKUP for backups and restore, SYSDG  for administration of Data Guard.

Along the same lines, We have new privilege  PURGE DBA_RECYCLEBIN that helps with implementing separation of duty. Users granted the PURGE DBA_RECYCLEBIN system privilege can purge the recyclebin for all users in the database using the PURGE DBA_RECYCLEBIN command.

Unified Auditing (Part-I)

Increased scrutiny of Organizations adherence to  regulatory guidance like SOX, Hippa & PCI DSS has made audit of   critical and privileged actions in the database more likely to be mandatory than optional. Therefore forcing database or software vendors to implement auditing with piecemeal approach. Oracle is no different with various  ways to audit  & various  locations  to store them in various formats.  For example some of the auditing options available with Oracle 11g are mandatory auditing, standard database auditing , Data Vault auditing & FGA and so on ….

Staying complaint is expensive , time consuming and complex. Some of the complexities involved with auditing has been simplified with unified auditing available with Oracle12c. In this blog , we will discuss about Unified auditing in 4 part series. Here is the 1st part.

So what is unified auditing ?
Unified Auditing is selective and effective auditing  of database actions to a single unified audit trail  inside of the database using policies and conditions.  Most of unified auditing features are implemented using DBMS_AUDIT_MGMT or Oracle Grid Control. DBMS_AUDIT_MGMT is not a new to some/most of us as it is available with Oracle11g. It is enhanced in Oracle12c to support unified auditing.  Unified auditing supports auditing of following components.

  1. FGA
  2. Real Application Security
  3. RMAN
  4. Database Vault
  5. Oracle Label Security
  6. Data Mining
  7. Data Pump
  8. SQL*Loader

My next question why do we need a unified auditing? What is wrong with existing auditing?  Why couldn’t we audit everything in the database and get done with it and make our auditors happy. What actually stopped us from auditing everything in the database? The main reasons were performance impact caused by auditing followed by security.  Now we have the burden on us to prove that all our audit is safe, correct and not tampered with. And finally where do I store all the audit data ; Getting additional storage for database itself is cumbersome…. Now you need space to store all of audit data.

Let us look at how unified auditing measures  itself against the 3 metrics: Performance, Security & Storage

Performance

As per Oracle, Unified auditing  results in  negligible overhead. Why ? Because it  is implemented in SGA using named Queues. Dequeing is continuous with each audit client (Example: RMAN, Oracle Label Security) having two separate queues so that the client can continue to write to the second queue while the first queue is being dequeued to database table by background process GEN0. if needed audit Data can also be manually flushed to the database using procedure FLUSH_UNIFIED_AUDIT_TRAIL.

Example: EXECUTE DBMS_AUDIT_MGMT.FLUSH_UNIFIED_AUDIT_TRAIL;

As usual we have the choice of synchronous and asynchronous dequeuing. In  unified audit terms, they are called as Immediate write mode and queued write mode.

Immediate-Write mode

Audit records are immediately written to the trail and there is a performance overhead associated with this approach

Example: EXECUTE DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_PROPERTY (DBM_MS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED,DBMS_AUDIT_MGMT.AUDIT_TRAIL_WRITE_MODE,  DBMS_AUDIT_MGMT.AUDIT_TRAIL_IMMEDIATE_MODE);

Queued-Write mode(Default)

The named queue in the SGA is  dequeued periodically. In this mode , data could be lost after an instance crash and therefore its a trade-in  for data loss for performance. How cruel …. never can have everything

Example: EXECUTE DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_PROPERTY (DBM_MS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED,DBMS_AUDIT_MGMT.AUDIT_TRAIL_WRITE_MODE, DBMS_AUDIT_MGMT.AUDIT_TRAIL_QUEUED_MODE);

Summary: Performance is better mainly because of  queued-write mode architecture; Using immediate-write mode will cause performance impact but not as significant pre12c auditing.

 

 

Let us look at other metrics in Part-II of this series. I will post the link once  I am ready.

Collecting stats during bulk load

Starting with Oracle 12c, the default behavior of database is to gathers statistics during bulk loads like CTAS or INSERT SELECT.  However for some reason , if you want to go back to pre-12c behavior , you can do so with a new hint NO_GATHER_OPTIMIZER_STATISTICS hint.

See illustration of examples comparing Oracle11g and Oracle12c

Oracle 11g

SQL> SELECT * FROM V$VERSION;

BANNER
——————————————————————————–
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 – 64bit Production
PL/SQL Release 11.2.0.2.0 – Production
CORE    11.2.0.2.0      Production
TNS for Linux Version 11.2.0.2.0 – Production
NLSRTL Version 11.2.0.2.0 – Production

SQL> CREATE TABLE MY_OBJECTS_NOSTATS AS SELECT * FROM MY_OBJECTS;

Table created.

SQL> CREATE TABLE MY_OBJECTS_STATS AS SELECT /*+ GATHER_OPTIMIZER_STATISTICS */  * FROM MY_OBJECTS; ==> Database is 11g, Hint is ignored

Table created.

SQL>  CREATE TABLE MY_OBJECTS_NOSTATS_HINT  AS SELECT /*+ NO_GATHER_OPTIMIZER_STATISTICS */  * FROM MY_OBJECTS; ==> Database is 11g, Hint is ignored

Table created.

SQL> SELECT TABLE_NAME, LAST_ANALYZED , NUM_ROWS, BLOCKS FROM DBA_TABLES WHERE TABLE_NAME LIKE ‘MY_OBJECTS%’; ==> Database is 11g, Stats are not gathered for all tables

TABLE_NAME                     LAST_ANAL   NUM_ROWS     BLOCKS
—————————— ——— ———- ———-
MY_OBJECTS
MY_OBJECTS_STATS
MY_OBJECTS_NOSTATS
MY_OBJECTS_NOSTATS_HINT

Oracle12c

SQL> SELECT BANNER FROM V$VERSION;

BANNER
——————————————————————————–
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 – 64bit Production
PL/SQL Release 12.1.0.1.0 – Production
CORE    12.1.0.1.0      Production
TNS for Linux: Version 12.1.0.1.0 – Production
NLSRTL Version 12.1.0.1.0 – Production

SQL> CREATE TABLE MY_OBJECTS_NOSTATS AS SELECT * FROM MY_OBJECTS;

Table created.

SQL> CREATE TABLE MY_OBJECTS_STATS AS SELECT /*+ GATHER_OPTIMIZER_STATISTICS */  * FROM MY_OBJECTS; ==> Default behavior ;hint is of no help here.

Table created.

SQL> CREATE TABLE MY_OBJECTS_NOSTATS_HINT  AS SELECT /*+ NO_GATHER_OPTIMIZER_STATISTICS */  * FROM MY_OBJECTS; ==> Hint instructing Oracle not to collect Stats

Table created.

SQL> SET LINESIZE 200
SQL> COLUMN TABLE_NAME FOR A40
SQL> SELECT TABLE_NAME, LAST_ANALYZED , NUM_ROWS, BLOCKS FROM DBA_TABLES WHERE TABLE_NAME LIKE ‘MY_OBJECTS%’; ==> Stats are gathered for all tables except table MY_OBJECTS_NOSTATS_HINT which was created with NO_GATHER_OPTIMIZER_STATISTICS hint

TABLE_NAME                               LAST_ANAL   NUM_ROWS     BLOCKS
—————————————————————————————————————
MY_OBJECTS                                26-AUG-13      90908       1561
MY_OBJECTS_NOSTATS                26-AUG-13      90908       1561
MY_OBJECTS_STATS                    26-AUG-13      90908       1561
MY_OBJECTS_NOSTATS_HINT

4 rows selected.

SQL>

TABLE ACCESS BY INDEX ROWID BATCHED

TABLE ACCESS BY INDEX ROWID BATCHED is new execution plan operation that helps improve performance.  It is generally used for range ( > or < ) queries.  For this new operation, Oracle selects few  ROWIDs from the index and then try to access the rows in blocks. This significantly reduces the number of times Oracle must access the blocks thereby improving performance.

Sample Execution plan

Execution Plan
———————————————————-
Plan hash value: 525114061

—————————————————————————————————————————-
| Id  | Operation                                                   | Name     | Rows   | Bytes   | Cost (%CPU)| Time
—————————————————————————————————————————–
|   0 | SELECT STATEMENT                                   |                 |    10  |  3020  |     3   (0)| 00:00:01
|*  1 |  COUNT STOPKEY                                      |                 |          |           |              |
|   2 |   TABLE ACCESS BY INDEX ROWID BATCHED | EMP          | 28012 |  8261K|     3   (0)| 00:00:01
|*  3 |    INDEX RANGE SCAN                                | EMP_SAL_I |          |           |     2   (0)| 00:00:01
—————————————————————————————————————————–

Oracle12c:Temporal Validity (Part-1)

You will really appreciate this feature if you have called some of the cable companies to terminate your cable connection.   This is the response you generally get to hear ” Please call us on so and so date; I cannot update the system now”. Hopefully some of those companies can use these feature now.

Temporal Validity  associates one or more valid time dimensions to your data making it visible or invisible  depending on the time as set by the user. If adopted , may be with this feature, you can change your address or phone number months in advance, don’t have to wait for the last minute.

So how do you implement it?

PERIOD FOR clause of  CREATE/ALTER table command is used to implement temporal validity.

The syntax for PERIOD FOR clause is as below.

PERIOD FOR dimension_name [( start_time_column , end_time_column)]

You can explicitly  specify  columns “start_time_column”  & “end_time_column”  and when specified  they must be existing columns in the table.

If  “start_time_column”  & “end_time_column”   is skipped , Oracle will create  two invisible columns  dimension_name_start & dimension_name_end for time dimensions. The invisible columns cane be used in INSERT  & UPDATE statement like like any defined table columns.

 

start_time_column —–end_time_column ——-Result
NULL NULL Valid for all Times
NULL Value Valid for all times before  end_time_column values
Value NULL Valid for all times after start_time_column values
Value Value Valid for all times between start_time_column and end_time_column values

See examples below for creating tables

In the 1st example below, I am creating a time dimension using columns of the table itself. Two columns SERVICE_START and SERVICE_END  specifies the time dimension for the row.

SQL> CREATE TABLE CUSTOMER_1
2  (
3  CUST_NO    NUMBER(12) PRIMARY KEY,
4  CUST_NAME  VARCHAR2(20) NOT NULL,
5  CUST_ADDR  VARCHAR2(40),
6  SERVICE_START      TIMESTAMP,
7  SERVICE_END        TIMESTAMP,
8  PERIOD FOR CUST_HIST_TIME(SERVICE_START, SERVICE_END)
9  );

Table created.

In the 2nd example below, I am creating the  table  without specifying the time dimensions for the data. In such case Oracle will create 2 invisible columns CUST_HIST_TIME_START & CUST_HIST_TIME_END that can be used  almost like any other regular table column.

Not: CUST_HIST_TIME is user defined value for time dimension specified with PERIOD FOR clause.

SQL> CREATE TABLE CUSTOMER_2
2  (
3  CUST_NO    NUMBER(12) PRIMARY KEY,
4  CUST_NAME  VARCHAR2(20) NOT NULL,
5  CUST_ADDR  VARCHAR2(40),
6  PERIOD FOR CUST_HIST_TIME ==> Invisible start and end columns are created
7  );

Table created.

Inserting data into temporal tables with invisible columns

SQL> INSERT INTO CUSTOMER_2 (CUST_NO, CUST_NAME, CUST_ADDR, CUST_HIST_TIME_START , CUST_HIST_TIME_END)
2  VALUES (1,’SHAN’,’1235 RIVER RD VA 20155′,SYSDATE, SYSDATE+90);

1 row created.

SQL> INSERT INTO CUSTOMER_2 (CUST_NO, CUST_NAME, CUST_ADDR, CUST_HIST_TIME_START , CUST_HIST_TIME_END)
2   VALUES (2,’JOE’,’78628 LAKE ST IL 60148′,SYSDATE , SYSDATE+120);

1 row created.

Click here for Part-2 : Work-in-progress