Story of A NoSQL “IfDB” Project Suspended in 2004

> for Japanese Pages

Story of A NoSQL “IfDB” Project Suspended in 2004
1. A story over 14 years ago
The keyword NoSQL was first proposed in 1998.
However, it did not penetrate at all.
And in early 2009, NoSQL gradually began to penetrate by re-proposing a person.

The other day, I was sorting out my own source code, I stayed in my eyes in the source code of the last update November 2004.
The source code is a NoSQL scratch source code called “Document Oriented DB” in recent years.

NoSQL's first proposal was in 1998.
We were developing from 2003 to 2004.
NoSQL began to penetrate the world in 2009.
The year of NoSQL's popularization is much later.
In short, it is a project more than 5 years ago from the re-proposal of NoSQL.
That is a story of over 15 years ago.

What I thought about this source code again is that “such technical innovation approach is not wrong.”
And one technology I developed at that time is also useful for my AI research now.
I want to say “Don't give up” to such innovative technicians.
And I would like to thank the engineers that developed NoSQL.


2. About NoSQL Project IfDB
Well, let me briefly outline this project.
This project name has been changed several times as follows.

FileDB -> IniDB -> IfDB
At that time, we were not conscious of the word NoSQL at all.
Rather, we probably didn't know NoSQL.
The CRUD processing of data is flow as follows.

Array -> Query Command -> Ini File -> Array of Result Set
We also implemented SQL-like syntaxes.
And what we emphasized was to enable direct editing of Ini File rather than via commands.
This is because at that time JSON was not generic.
These are still useful depending on applications but I have been busy with other projects and forgotten over the last ten years.
However, I have memories that the development speed and the improvement cycle have increased very much using these method.


3. The reason why I developed Ifdb
The reason why I developed this technology 14 years ago.
I saw this source code after a long time, but the concept is not much different from the current document-oriented DB.
14 years ago, I was troublesome that relational model design but I had doubts about waterfall-like data model design and schema restrictions.
This is because the scalability of the project is compromised by staticizing the relational model and schema.
At that time, all the DBA's arranged their mouths and said:

“It's because the design of the data model is bad.”
“Study the data model more.”

However, the more I learned data models, the more my discomfort grew.
I think that I probably wanted to do agile development at that time.
At that time I probably did not even know about agile development.
The initial proposal time for agile development is also in the 1990's.


4. Conclusion
The de facto standard of technology is a very important factor in business.
However, the number of engineers who do not think about the simple thing “why the de facto standard is important” has increased.
They arrange their mouths and make excuses like this:

“Not a de facto standard = evil”
“We don't need scratch development.”

You don't forget what you are developing in scratch development products of me or someone.

Both Sustaining Innovation and Disruptive Innovation are important.

It is always proved by the times.

Operational Notifications Exception Design (ONED) for Web Applications by DevOps

> to Japanese Pages
1. Abstract
With recent intensification of speed competition of IT service development, in this post I will explain the new exception design (ONED) approach of the Web application based on DevOps aiming at improving the improvement cycle of development speed, quality and opportunity loss rate.



2. Representation specific to ONED
# ONED
Define “Operational Notifications Exception Design” as ONED.

# Event Exception Class
Define event-based exception classes such as SyntaxException and ArgumentException as event exception classes.

# Exception Trigger
Define throw syntax of exception as exception trigger.

# Exception Instance
If only an instance of an exception class, define it as an exception instance.

# Exception Captor
Define catch, rescue, except syntax of exception as exception captor.


3. Scope of this Article
The scope of this article is listed below.

Since this article is intended to explain the concept of ONED, it does not mention the exception mechanism.
To avoid misunderstanding, this article is limited for web applications.
This article excludes general purpose libraries, general purpose tools and local applications from subjects.
Because the syntax may differ depending on the development language, you should replace it as appropriate as your development language.
This article does not mention Checked Exception or Unchecked Exception.
In this article, I write only the main points because I do not intend to write 8 pages as in academic papers.
We all want to think about the meaning of the word of Exception occasionally, but let's give it up as each language specification.

4. Introduction
What are the functional requirements you will undertake at the beginning of the web application implementation?
I will undertake exception design and the implementation.
This is because projects whose exception design and implementation are delayed will be forced to refactor all source code in the latter part of the implementation and will have an adverse effect on everything.
If exception design and implementation are completed, I can think that 50% of Web application implementation was completed.

Therefore, the most important reason is to raise improvement cycle of development speed, quality and opportunity loss.
This is because it is indispensable to improve development speed and quality control in order to survive in the market as the competition of development speed of IT services in recent years intensifies.
In recent years, the concept of various service development has spread from marketing methods such as Scrum development, DevOps, Growth Hack/Marketing, Lean Development, and DCPA (Not PDCA) to management methods.
The common thing to these is the objective of promoting service growth and improvement cycle.
Optimization of exception design is also an important factor that can contribute to these improvements if these are the objectives.

So, what should we do with exception design for these purposes?
In this post, I would like to touch on this theme from ONED's approach.

By the way, when I lecture exception design to members, I often receive a question from members.

“I don't know the difference between Exception and Error.”

This is a very important question in exception design.
For this question, see the previous article: Exception and Error.
The most important point in exception design is that you can properly organize errors and exceptions within yourself.


5. Recommendation
5-1. Elements of Exception Class
The elements of the recommended exception class by ONED are as follows.
(You should replace the syntax according to the development language.)

# Example.1:
InternalCriticalException("messages: HTTP_STATUS_CODE")

# Example.2:
FooException("InternalCritical: messages: HTTP_STATUS_CODE")

I explain the above points and usefulness in order.
For convenience of explanation, I will explain it using Example 1.


5-2. the Point of Responsibility Division
# Example.1:
InternalCriticalException("messages: HTTP_STATUS_CODE")

The prefix of the exception class name in Example 1 means the point of responsibility division.
In ONED, examples of recommended division of responsibility points are as follows.

# Internal
Internal System: Responsibility of The Exception Occurrence Host
For example, in this case Web server.

# External
External System:  Responsibility of External System
For example, external systems such as Google APIs etc.

# Medial
Medial System: Responsibility of Owned Resources
For example, DB servers viewed from a web server etc.

# Terminal
Client System:  Responsibility of Client System
For example, web clients, API clients and user devices etc.

In this way, the responsibility division point clarifies the occurrence responsibility of the exception.
Why do we need to clarify the responsibility for exception occurrence?
In the first place, clarifying the responsibility of occurrence of system failure is the basis of initial response.
This is because clarifying the occurrence responsibility in the initial response improves the recovery speed.

And ONED is based on the idea that emphasizes the initial response of operations recovery.
If you are a top engineer with rich application development/operations experience, you will understand these usefulness only with this explanation.

In short, it is important for developers to code with operations awareness.
However, the above actions are all cases of high importance exceptions.
In next section, I explain the importance lavel of exception.


5-3. Exception Level (Importance)
# Example.1:
InternalCriticalException("messages: HTTP_STATUS_CODE")

The second word above represents the error level.
Normally, error level such as logging often indicates importance, but it is the same in ONED.
However, I occasionally receive the following questions:

“Which level of importance should I use?”

“Which exceptions are important and which exceptions are unimportant?”

“In order to master these, I need a lot of experience like you.”

It's not like that.
Let's think about the service operations again.
For example, suppose you are implementing an exception code like "FileNotFound".
In that case, if this exception occured after service release, should you receive emergency mail as the development manager?
If it is a minor exception that can continue processing by automatic recovery, such emergency mail notifications are only noisy.
If you e-mailed every exception, we will lose sight of important notifications or no one will care about the notification.
The importance level of ONED is only the level of the definition of automatic processing that must be executed during operations.
In other words, when that exception occured, what do you want to execute handle with error handler during service operations?

Do you want logging?

Do you want display the error message to end users?

Do you want continue processing?

Do you want receive the emergency mail?

Do you want receive the emergency tel?

Just thinking about these things.
If it was excessive notifications, you just adjust it after release.
Exception Level = Importance = Processing Content at Operations
See the table below:

Exception Level Example:

    

        
Level
        
Log
        
Display
        
Mail
        
Tel
        
Abort
    
    

        
Info
        
o
        
x
        
x
        
x
        
x
    
    

        
Notice
        
o
        
o
        
x
        
x
        
x
    
    

        
Warning
        
o
        
o
        
x
        
x
        
o
    
    

        
Pratial
        
o
        
o
        
o
        
x
        
Partial
    
    

        
Error
        
o
        
o
        
o
        
x
        
x
    
    

        
Critical
        
o
        
o
        
o
        
o
        
o
    
    

        
Fatal
        
o
        
o
        
o
        
o
        
Shutdown
    

In this way, the importance and the processing content are merely linked.
Also, the exception level should not be superficial and should be practical for service operations.
Therefore, we should be able to change the above setting for each service.
In short, the viewpoint of operations is important for exception coding.


5-4. Pros and Cons of Emergency Level
However, we sholud be careful.
That is, exception level of ONED must be importance level and must not be emergency level.
This is because ONED thinks that urgency at the time of exception of Web application should be overall judgement.
The overall judgment is a comprehensive judgment of the service operations status, the resource situation, the extent of influence to the user, the consistency of the data, and the like.
This is because even under the same exception, depending on these circumstances, judgment may be given not to respond urgently.
And because the defined word "emergency" at the development phase becomes meaningless at the time of operations recovery.
Urgency should be judged in the operations phase.
Emergency level is used even for the log level recommended by the Web-only language like PSR-3, and descriptions related to urgency are arranged side by side.
Defining such urgency only in the development stage is evidence that it is development driven and it is also the cause of vertical development.


5-5. HTTP Status Code
In the case of exception handling of a Web application, the trigger should convey the HTTP status code so that the exception captor can properly return the HTTP status code to the HTTP client.

# PHP
throw new Exception('message', 500);

# Ruby
raise Exception.new("messages: code")

This is because the top-level captor can not accurately determine which status code should be returned to HTTP clients.
This is particularly important for Web API servers that have to handle with status codes sensitive.


5-6. Continuation and Interruption Processing of Exception
ONED is not concept like “exception = interruption”
I feel that the stereotype like the one above is often found in web application engineers.
Even for Web applications, continuation of exceptions is a very important factor.
So, in next section, I will explain the way of thinking of Exception Continuation Processing and examples of the implementation.


5-7. Implementation Examples of Continuation Processing
Continuation processing of ONED is to execute necessary recovery processing and notification processing after exception occurrence and return to normal processing, and there is no matter how to realize it.
Examples of implementation is shown below.

# Retry Mechanism
How to retry an exception block after continuation processing of exception using a retry mechanism like Ruby or Python package.

# Goto Mechanism
A method of returning processing up to a label declared near an exception trigger after continuation processing of an exception using a goto mechanism like C family, PHP, golang, Python package.

# Exception Block
A way to make use of exception blocks variously for recovery/continuation processing.
(In the case of Java, checked exception)

# Wrapping Recovery/Continuation in Exception Class
How to wrap recovery/continuation processing within an exception class.
The misunderstanding "Instance and Throw" by some engineers will be described later.

# Exceptions are captured at the top level.
if (! is_file("/dir/file.txt")) {
    // Instance
    $FileNotFound = new FileNotFoundException("Could not find the file.", 500);
    // Recovery Processing
    if (! $FileNotFound->makeFile("/dir/file.txt")) {
        // Interruption Processing
        throw $FileNotFound;
    }
    // Continuation Processing
    $FileNotFound->log("Notice");
    $FileNotFound->mail("Notice");
}
// Standard Processing


# Wrapping Continuation/Interruption in Exception Class
How to wrap continuation/interruption processing within an exception class.
# Exceptions are captured at the top level.
if (! is_file("/dir/file.txt")) {
    // try Recovery Processing
    if (! touch("/dir/file.txt")) {
        // Interruption Processing
        new InternalCriticalException("Could not create the file.", 500);
    }
    // Continuation Processing
    new InternalNoticeException("Could not find the file.", 500);
}
// Standard Processing
↓
class InternalCriticalException extends X
{
    const LOG = true;
    const DISPALY = true;
    const MAIL = true;
    const ABORT = true;
    const TEL = true;
    const SHUTDONW = false;
}
↓
class X extends Exception
{
    public function __construct()
    {
        parent::__construct();
        //...
    }

    protected function _log()
    {
        //...
    }

    protected function _display()
    {
        //...
    }

    protected function _mail()
    {
        //...
    }

    protected function _tel()
    {
        //...
    }

    protected function _abort()
    {
        throw $this;
    }

    protected function _shutdown()
    {
        //...
    }
}


5-7. Last Captor
In Web application development by ONED, we think that you should control all exceptions of unexpected exceptions and expected exceptions by using top level captor.


6. Pros and Cons of Exception Handling
6-1. Misunderstanding of The Throw Keyword
throw ≠ instance
Very rarely, there are programmers who have a strange misunderstanding about the “throw” keyword of exception.
For example, a subclass that inherits a throwable class is a very strange misunderstanding that you must throw at the same time as the class instance.
This is typical of catalog engineers who do not understand exceptions at all.
Do you know the meaning of the catalog engineer?
It is a negative evaluation that was frequently used in automobile manufacturing industry like Toyota.
If there are such constraints, we are not allowed to re-throw an instanced object?
The “throw” keyword is simply a syntax that throws the object, its timing is arbitrary.
Also, it is arbitrary whether to throw the object.
throw new FooException("messages");
Such a syntax just throws an object at the same time as an instance.
There are no such restrictions in C languages, Python, Java, Ruby, Perl, PHP, Javascript and any other languages.
Such stubborn justification is very nonsense.


6-2. Non-Local Exits
Non-Local Exsits ≠ Error Destoroy
I sometimes see a negative article as below.
“The coding as below is bad.”:
// Java Example:
try {
    // processing
    // ...
} catch (FooException e) {}

Certainly it is the worst code.
If it is the purpose of destroying exceptions and errors it is guilty.
We should feedback if he forgets to code.
If he is inexperienced in coding for non-local exits, we should tell him the best way by development language.
The one that should really be improved is the unfounded stereotype that "catch block is empty = evil".


6-3. Event Exception Class
FileNotFoundException
ONED defines exception classes such as FileNotFoundException and AugmentException as event exception classes.
What we have to note is that there are Exception to be Continued and Exception to be Suspended even with the same Exception of FileNotFound.
We also have to be careful that the class name perspective is development driven.
Personally, I recommend naming the operations viewpoint.


7. Conclusion
In this way, ONED is an exception design based on DevOps.
In fact, for all Web services that introduced ONED, the development speed, quality improvement and service growth cycle dramatically improved.
This article is a post that modified my research paper for practical use.