How-To Tutorials

If you are running a PostgreSQL setup, there are basically two major methods to perform backups: Logical dumps (extract an SQL script representing your data) Transaction log shipping The idea behind transaction log shipping is to archive binary changes made to the database. Most people claim that transaction log shipping is the only real way to do backups. However, in my opinion, this is not necessarily true. Many people rely on pg_dump to simply extract a textual representation of the data. Interestingly, pg_dump is also the oldest method of creating a backup and has been around since the very early days of the PostgreSQL project (transaction log shipping was added much later). Every PostgreSQL administrator will become familiar with pg_dump sooner or later, so it is important to know how it really works and what it does. This article is an excerpt taken from the book Mastering PostgreSQL 11 - Second Edition by Hans-Jürgen Schönig. In this book, you will learn the approach to get to grips with advanced PostgreSQL 11 features and SQL functions, master replication and failover techniques, configure database security and more. In this article, you will learn the process of partially dumping data, restoring backups, saving global data and much more. Running pg_dump The first thing we want to do is create a simple textual dump: [hs@linuxpc ~]$ pg_dump test > /tmp/dump.sql This is the most simplistic backup you can imagine. Basically, pg_dump logs in to the local database instance, connects to a database test, and starts to extract all the data, which will then be sent to stdout and redirected to the file. The beauty, here, is that the standard output gives you all the flexibility of a Unix system. You can easily compress the data using a pipe or do whatever you want to do with it. In some cases, you might want to run pg_dump as a different user. All PostgreSQL client programs support a consistent set of command-line parameters to pass user information. If you just want to set the user, use the -U flag as follows: [hs@linuxpc ~]$ pg_dump -U whatever_powerful_user test > /tmp/dump.sql The following set of parameters can be found in all PostgreSQL client programs: ... Connection options: -d, --dbname=DBNAME database to dump -h, --host=HOSTNAME database server host or socket directory -p, --port=PORT database server port number -U, --username=NAME connect as specified database user -w, --no-password never prompt for password -W, --password force password prompt (should happen automatically) --role=ROLENAME do SET ROLE before dump ... You can just pass the information you want to pg_dump, and if you have enough permissions, PostgreSQL will fetch the data. The important thing here is to see how the program really works. Basically, pg_dump connects to the database and opens a large repeatable read transaction that simply reads all the data. Remember, a repeatable read ensures that PostgreSQL creates a consistent snapshot of the data, which does not change throughout the transactions. In other words, a dump is always consistent—no foreign keys will be violated. The output is a snapshot of data as it was when the dump started. Consistency is a key factor here. It also implies that changes made to the data while the dump is running won't make it to the backup anymore. A dump simply reads everything—therefore, there are no separate permissions to be able to dump something. As long as you can read it, you can back it up. Passing passwords and connection information If you take a close look at the connection parameters shown in the previous section, you will notice that there is no way to pass a password to pg_dump. You can enforce a password prompt, but you cannot pass the parameter to pg_dump using a command-line option. The reason for this is simply because the password might show up in the process table and be visible to other people. The question now is, if pg_hba.conf, which is on the server, enforces a password, how can the client program provide it? There are various means of doing this. Some of them are as follows: Making use of environment variables Making use of .pgpass Using service files In this section, we will learn about all three methods. Extracting subsets of data Up until now, we have seen how to dump an entire database. However, this is not what we might wish for. In many cases, we just want to extract a subset of tables or schemas.  Fortunately, pg_dump can help us do that while also providing a number of switches: -a: It only dumps the data and does not dump the data structure -s: It dumps the data structure but skips the data -n: It only dumps a certain schema -N: It dumps everything but excludes certain schemas -t: It only dumps certain tables -T: It dumps everything but certain tables (this can make sense if you want to exclude logging tables and so on) Partial dumps can be very useful in order to speed things up considerably. Handling various formats So far, we have seen that pg_dump can be used to create text files. The problem here is that a text file can only be replayed completely. If we have saved an entire database, we can only replay the entire thing. In most cases, this is not what we want. Therefore, PostgreSQL has additional formats that offer more functionality. At this point, four formats are supported: -F, --format=c|d|t|p output file format (custom, directory, tar, plain text (default)) We have already seen plain, which is just normal text. On top of that, we can use a custom format. The idea behind a custom format is to have a compressed dump, including a table of contents. Here are two ways to create a custom format dump: [hs@linuxpc ~]$ pg_dump -Fc test > /tmp/dump.fc [hs@linuxpc ~]$ pg_dump -Fc test -f /tmp/dump.fc In addition to the table of contents, the compressed dump has one more advantage. It is a lot smaller. The rule of thumb is that a custom format dump is around 90% smaller than the database instance you are about to back up. Of course, this is highly dependent on the number of indexes, but for many database applications, this rough estimation will hold true. Once the backup is created, we can inspect the backup file: [hs@linuxpc ~]$ pg_restore --list /tmp/dump.fc ; ; Archive created at 2018-11-04 15:44:56 CET ; dbname: test ; TOC Entries: 18 ; Compression: -1 ; Dump Version: 1.12-0 ; Format: CUSTOM ; Integer: 4 bytes ; Offset: 8 bytes ; Dumped from database version: 11.0 ; Dumped by pg_dump version: 11.0 ; ; Selected TOC Entries: ; 3103; 1262 16384 DATABASE - test hs 3; 2615 2200 SCHEMA - public hs 3104; 0 0 COMMENT - SCHEMA public hs 1; 3079 13350 EXTENSION - plpgsql 3105; 0 0 COMMENT - EXTENSION plpgsql 187; 1259 16391 TABLE public t_test hs ... Note that pg_restore --list will return the table of contents of the backup. Using a custom format is a good idea as the backup will shrink in size. However, there's more; the -Fd command will create a backup in the directory format. Instead of a single file, you will now get a directory containing a couple of files: [hs@linuxpc ~]$ mkdir /tmp/backup [hs@linuxpc ~]$ pg_dump -Fd test -f /tmp/backup/ [hs@linuxpc ~]$ cd /tmp/backup/ [hs@linuxpc backup]$ ls -lh total 86M -rw-rw-r--. 1 hs hs 85M Jan 4 15:54 3095.dat.gz -rw-rw-r--. 1 hs hs 107 Jan 4 15:54 3096.dat.gz -rw-rw-r--. 1 hs hs 740K Jan 4 15:54 3097.dat.gz -rw-rw-r--. 1 hs hs 39 Jan 4 15:54 3098.dat.gz -rw-rw-r--. 1 hs hs 4.3K Jan 4 15:54 toc.dat One advantage of the directory format is that we can use more than one core to perform the backup. In the case of a plain or custom format, only one process will be used by pg_dump. The directory format changes that rule. The following example shows how we can tell pg_dump to use four cores (jobs): [hs@linuxpc backup]$ rm -rf * [hs@linuxpc backup]$ pg_dump -Fd test -f /tmp/backup/ -j 4 The more objects in our database, the more of a chance there is for a potential speedup. Replaying backups Having a backup is pointless unless you have tried to actually replay it. Fortunately, this is easy to do. If you have created a plain text backup, simply take the SQL file and execute it. The following example shows how that can be done: psql your_db < your_file.sql A plain text backup is simply a text file containing everything. We can always simply replay a text file. If you have decided on a custom format or directory format, you can use pg_restore to replay the backup. Additionally, pg_restore allows you to do all kinds of fancy things such as replaying just part of a database and so on. In most cases, however, you will simply replay the entire database. In this example, we will create an empty database and just replay a custom format dump: [hs@linuxpc backup]$ createdb new_db [hs@linuxpc backup]$ pg_restore -d new_db -j 4 /tmp/dump.fc Note that pg_restore will add data to an existing database. If your database is not empty, pg_restore might error out but continue. Again, -j is used to throw up more than one process. In this example, four cores are used to replay the data; however, this only works when more than one table is being replayed. If you are using a directory format, you can simply pass the name of the directory instead of the file. As far as performance is concerned, dumps are a good solution if you are working with small or medium amounts of data. There are two major downsides: We will get a snapshot, so everything since the last snapshot will be lost Rebuilding a dump from scratch is comparatively slow compared to binary copies because all of the indexes have to be rebuilt Handling global data In the previous sections, we learned about pg_dump and pg_restore, which are two vital programs when it comes to creating backups. The thing is, pg_dump creates database dumps—it works on the database level. If we want to back up an entire instance, we have to make use of pg_dumpall or dump all of the databases separately. Before we dig into that, it makes sense to see how pg_dumpall works: pg_dumpall > /tmp/all.sql Let's see, pg_dumpall will connect to one database after the other and send stuff to standard out, where you can process it with Unix. Note that pg_dumpall can be used just like pg_dump. However, it has some downsides. It does not support a custom or directory format, and therefore does not offer multicore support. This means that we will be stuck with one thread. However, there is more to pg_dumpall. Keep in mind that users live on the instance level. If you create a normal database dump, you will get all of the permissions, but you won't get all of the CREATE USER statements. Those globals are not included in a normal dump—they will only be extracted by pg_dumpall. If we only want the globals, we can run pg_dumpall using the -g option: pg_dumpall -g > /tmp/globals.sql In most cases, you might want to run pg_dumpall -g, along with a custom or directory format dump to extract your instances. A simple backup script might look such as this: #!/bin/sh BACKUP_DIR=/tmp/ pg_dumpall -g > $BACKUP_DIR/globals.sql for x in $(psql -c "SELECT datname FROM pg_database WHERE datname NOT IN ('postgres', 'template0', 'template1')" postgres -A -t) do pg_dump -Fc $x > $BACKUP_DIR/$x.fc done It will first dump the globals and then loop through the list of databases to extract them one by one in a custom format. To summarize, in this article, we learned about creating backups and dumps in general. To know more about streaming replication and binary backups, check out our book Mastering PostgreSQL 11 - Second Edition. Handling backup and recovery in PostgreSQL 10 [Tutorial] Understanding SQL Server recovery models to effectively backup and restore your database Saving backups on cloud services with ElasticSearch plugins

Pretrained models are used in the following two popular ways when building new models or reusing them: Using a pretrained model as a feature extractor Fine-tuning the pretrained model This article is an excerpt taken from the book Hands-on transfer learning with Python. This book covers the process of setting up of DL environment and talks about various DL architectures, including CNN, LSTM, and capsule networks and more. In this article, we will leverage a pre-trained model that is basically an expert in the computer vision domain and renowned for image classification and categorization. The pretrained model we will be using in this article is the popular VGG-16 model, created by the Visual Geometry Group at the University of Oxford, which specializes in building very deep convolutional networks for large-scale visual recognition. You can find out more about it on the official website of robots. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) evaluates algorithms for object detection and image classification at large scale and their models have often secured the first place in this competition. A pretrained model like the VGG-16 is an already trained model on a huge dataset (ImageNet) with a lot of diverse image categories. Considering this fact, the model should have learned a robust hierarchy of features, which are spatial, rotation, and translation invariant, as we have discussed before with regard to features learned by CNN models. Hence, the model, having learned a good representation of features for over a million images belonging to 1,000 different categories, can act as a good feature extractor for new images suitable for computer vision problems. These new images might never exist in the ImageNet dataset or might be of totally different categories, but the model should still be able to extract relevant features from these images, considering the principles of transfer learning. This gives us an advantage of using pretrained models as effective feature extractors for new images, to solve diverse and complex computer vision tasks, such as solving our cat versus dog classifier with fewer images, or even building a dog breed classifier, a facial expression classifier, and much more! Let's briefly discuss the VGG-16 model architecture before unleashing the power of transfer learning on our problem. Understanding the VGG-16 model The VGG-16 model is a 16-layer (convolution and fully connected) network built on the ImageNet database, which is built for the purpose of image recognition and classification. This model was built by Karen Simonyan and Andrew Zisserman and is mentioned in their paper titled Very Deep Convolutional Networks for Large-Scale Image Recognition. I recommend all interested readers to go and read up on the excellent literature in this paper.  The architecture of the VGG-16 model is depicted in the following diagram: You can clearly see that we have a total of 13 convolution layers using 3 x 3 convolution filters along with max-pooling layers for downsampling and a total of two fully connected hidden layers of 4,096 units in each layer followed by a dense layer of 1,000 units, where each unit represents one of the image categories in the ImageNet database. We do not need the last three layers since we will be using our own fully connected dense layers to predict whether images will be a dog or a cat. We are more concerned with the first five blocks, so that we can leverage the VGG model as an effective feature extractor. For one of the models, we will use it as a simple feature extractor by freezing all the five convolution blocks to make sure their weights don't get updated after each epoch. For the last model, we will apply fine-tuning to the VGG model, where we will unfreeze the last two blocks (Block 4 and Block 5) so that their weights get updated in each epoch (per batch of data) as we train our own model. We represent the preceding architecture, along with the two variants (basic feature extractor and fine-tuning) that we will be using, in the following block diagram, so you can get a better visual perspective: Thus, we are mostly concerned with leveraging the convolution blocks of the VGG-16 model and then flattening the final output (from the feature maps) so that we can feed it into our own dense layers for our classifier. Building our dataset To start with, we load up the following dependencies, including a utility module called utils, which is available in the utils.py file present in the code files. This is mainly used to get a visual progress bar when we copy images into new folders: import glob import numpy as np import os import shutil from utils import log_progress np.random.seed(42) Let's now load up all the images in our original training data folder as follows: files = glob.glob('train/*') cat_files = [fn for fn in files if 'cat' in fn] dog_files = [fn for fn in files if 'dog' in fn] len(cat_files), len(dog_files) Out [3]: (12500, 12500) We can verify with the preceding output that we have 12,500 images for each category. Let's now build our smaller dataset so that we have 3,000 images for training, 1,000 images for validation, and 1,000 images for our test dataset (with equal representation for the two animal categories): cat_train = np.random.choice(cat_files, size=1500, replace=False) dog_train = np.random.choice(dog_files, size=1500, replace=False) cat_files = list(set(cat_files) - set(cat_train)) dog_files = list(set(dog_files) - set(dog_train)) cat_val = np.random.choice(cat_files, size=500, replace=False) dog_val = np.random.choice(dog_files, size=500, replace=False) cat_files = list(set(cat_files) - set(cat_val)) dog_files = list(set(dog_files) - set(dog_val)) cat_test = np.random.choice(cat_files, size=500, replace=False) dog_test = np.random.choice(dog_files, size=500, replace=False) print('Cat datasets:', cat_train.shape, cat_val.shape, cat_test.shape) print('Dog datasets:', dog_train.shape, dog_val.shape, dog_test.shape) Cat datasets: (1500,) (500,) (500,) Dog datasets: (1500,) (500,) (500,) Now that our datasets have been created, let's write them out to our disk in separate folders, so that we can come back to them anytime in the future without worrying if they are present in our main memory: train_dir = 'training_data' val_dir = 'validation_data' test_dir = 'test_data' train_files = np.concatenate([cat_train, dog_train]) validate_files = np.concatenate([cat_val, dog_val]) test_files = np.concatenate([cat_test, dog_test]) os.mkdir(train_dir) if not os.path.isdir(train_dir) else None os.mkdir(val_dir) if not os.path.isdir(val_dir) else None os.mkdir(test_dir) if not os.path.isdir(test_dir) else None for fn in log_progress(train_files, name='Training Images'): shutil.copy(fn, train_dir) for fn in log_progress(validate_files, name='Validation Images'): shutil.copy(fn, val_dir) for fn in log_progress(test_files, name='Test Images'): shutil.copy(fn, test_dir) The progress bars depicted in the following screenshot become green once all the images have been copied to their respective directory: Pretrained CNN model as a feature extractor with image augmentation We will leverage the same data generators for our train and validation datasets that we used before. The code for building them is depicted as follows for ease of understanding: train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, horizontal_flip=True, fill_mode='nearest') val_datagen = ImageDataGenerator(rescale=1./255) train_generator = train_datagen.flow(train_imgs, train_labels_enc, batch_size=30) val_generator = val_datagen.flow(validation_imgs, validation_labels_enc, batch_size=20) Let's now build our deep learning model architecture. We won't extract the bottleneck features like last time since we will be training on data generators; hence, we will be passing the vgg_model object as an input to our own model: model = Sequential() model.add(vgg_model) model.add(Dense(512, activation='relu', input_dim=input_shape)) model.add(Dropout(0.3)) model.add(Dense(512, activation='relu')) model.add(Dropout(0.3)) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer=optimizers.RMSprop(lr=2e-5), metrics=['accuracy']) You can clearly see that everything is the same. We bring the learning rate slightly down since we will be training for 100 epochs and don't want to make any sudden abrupt weight adjustments to our model layers. Do remember that the VGG-16 model's layers are still frozen here and we are still using it as a basic feature extractor only: history = model.fit_generator(train_generator, steps_per_epoch=100, epochs=100, validation_data=val_generator, validation_steps=50, verbose=1) Epoch 1/100 100/100 - 45s 449ms/step - loss: 0.6511 - acc: 0.6153 - val_loss: 0.5147 - val_acc: 0.7840 Epoch 2/100 100/100 - 41s 414ms/step - loss: 0.5651 - acc: 0.7110 - val_loss: 0.4249 - val_acc: 0.8180 ... ... Epoch 99/100 100/100 - 42s 417ms/step - loss: 0.2656 - acc: 0.8907 - val_loss: 0.2757 - val_acc: 0.9050 Epoch 100/100 100/100 - 42s 418ms/step - loss: 0.2876 - acc: 0.8833 - val_loss: 0.2665 - val_acc: 0.9000 We can see that our model has an overall validation accuracy of 90%, which is a slight improvement from our previous model, and also the train and validation accuracy are quite close to each other, indicating that the model is not overfitting. This can be reinforced by looking at the following plots for model accuracy and loss: We can clearly see that the values of train and validation accuracy are quite close to each other and the model doesn't overfit. Also, we reach 90% accuracy, which is neat! Let's save this model on the disk now for future evaluation on the test data: model.save('cats_dogs_tlearn_img_aug_cnn.h5') We will now fine-tune the VGG-16 model to build our last classifier, where we will unfreeze blocks 4 and 5, as we depicted at the beginning of this article. Pretrained CNN model with fine-tuning and image augmentation We will now leverage our VGG-16 model object stored in the vgg_model variable and unfreeze convolution blocks 4 and 5 while keeping the first three blocks frozen. The following code helps us achieve this: vgg_model.trainable = True set_trainable = False for layer in vgg_model.layers: if layer.name in ['block5_conv1', 'block4_conv1']: set_trainable = True if set_trainable: layer.trainable = True else: layer.trainable = False print("Trainable layers:", vgg_model.trainable_weights) Trainable layers: [<tf.Variable 'block4_conv1/kernel:0' shape=(3, 3, 256, 512) dtype=float32_ref>, <tf.Variable 'block4_conv1/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'block4_conv2/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'block4_conv2/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'block4_conv3/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'block4_conv3/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'block5_conv1/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'block5_conv1/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'block5_conv2/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'block5_conv2/bias:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'block5_conv3/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'block5_conv3/bias:0' shape=(512,) dtype=float32_ref>] You can clearly see from the preceding output that the convolution and pooling layers pertaining to blocks 4 and 5 are now trainable, and you can also verify which layers are frozen and unfrozen using the following code: layers = [(layer, layer.name, layer.trainable) for layer in vgg_model.layers] pd.DataFrame(layers, columns=['Layer Type', 'Layer Name', 'Layer Trainable']) The preceding code generates the following output: We can clearly see that the last two blocks are now trainable, which means the weights for these layers will also get updated with backpropagation in each epoch as we pass each batch of data. We will use the same data generators and model architecture as our previous model and train our model. We reduce the learning rate slightly since we don't want to get stuck at any local minimal, and we also do not want to suddenly update the weights of the trainable VGG-16 model layers by a big factor that might adversely affect the model: # data generators train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, horizontal_flip=True, fill_mode='nearest') val_datagen = ImageDataGenerator(rescale=1./255) train_generator = train_datagen.flow(train_imgs, train_labels_enc, batch_size=30) val_generator = val_datagen.flow(validation_imgs, validation_labels_enc, batch_size=20) # build model architecture model = Sequential() model.add(vgg_model) model.add(Dense(512, activation='relu', input_dim=input_shape)) model.add(Dropout(0.3)) model.add(Dense(512, activation='relu')) model.add(Dropout(0.3)) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer=optimizers.RMSprop(lr=1e-5), metrics=['accuracy']) # model training history = model.fit_generator(train_generator, steps_per_epoch=100, epochs=100, validation_data=val_generator, validation_steps=50, verbose=1) Epoch 1/100 100/100 - 64s 642ms/step - loss: 0.6070 - acc: 0.6547 - val_loss: 0.4029 - val_acc: 0.8250 Epoch 2/100 100/100 - 63s 630ms/step - loss: 0.3976 - acc: 0.8103 - val_loss: 0.2273 - val_acc: 0.9030 ... ... Epoch 99/100 100/100 - 63s 629ms/step - loss: 0.0243 - acc: 0.9913 - val_loss: 0.2861 - val_acc: 0.9620 Epoch 100/100 100/100 - 63s 629ms/step - loss: 0.0226 - acc: 0.9930 - val_loss: 0.3002 - val_acc: 0.9610 We can see from the preceding output that our model has obtained a validation accuracy of around 96%, which is a 6% improvement from our previous model. Overall, this model has gained a 24% improvement in validation accuracy from our first basic CNN model. This really shows how useful transfer learning can be. Let's observe the model accuracy and loss plots: We can see that accuracy values are really excellent here, and although the model looks like it might be slightly overfitting on the training data, we still get great validation accuracy. Let's save this model to disk now using the following code: model.save('cats_dogs_tlearn_finetune_img_aug_cnn.h5') Let's now put all our models to the test by actually evaluating their performance on our test dataset. In this article, we learned how to leverage pre-trained models for transfer learning and covered the various ways to use them, including as feature extractors, as well as fine-tuning. We saw the detailed architecture of the VGG-16 model and how to leverage the model as an efficient image feature extractor.  To know more about Pretrained CNN models check out our book Hands-On Transfer Learning with Python 5 types of deep transfer learning Neural Style Transfer: Creating artificial art with deep learning and transfer learning Dr. Brandon explains ‘Transfer Learning’ to Jon

At this point, it should not be a surprise that a number of major changes were announced in the Java ecosystem, some of which have the potential to force a reassessment of Java roadmaps and even vendor selection for enterprise Java users. Some of the biggest changes are taking place in the Upstream OpenJDK (Open Java Development Kit), which means that users will need to have a backup plan in place for the transition. Read on to better understand exactly what the changes Oracle made to Oracle JDK are, why you should care about them and how to choose the best next steps for your organization. For some quick background, OpenJDK began as a free and open source implementation of the Java Platform, Standard Edition, through a 2006 Sun Microsystems initiative. They made the announcement at the 2006 JavaOne event that Java and the core Java platform would be open sourced. Over the next few years, major components of the Java Platform were released as free, open source software using the GPL. Other vendors - like Red Hat - then got involved with the project about a year later, through a broad contributor agreement which covers the participation of non-Sun engineers in the project. This piece is by Rich Sharples, Senior Director of Product Management at Red Hat , on what Oracle ending free lifecycle support means, and what steps users can take now to prepare for the change. So what is new in the world of Java and JDK? Why should you care? Okay, enough about the history. What are we anticipating for the future of Java and OpenJDK? At the end of January 2019, Oracle officially ended free public updates to Oracle JDK for non-Oracle customer commercial users. These users will no longer be able to get updates without an Oracle support contract. Additionally, Oracle has changed the Oracle JDK license (BCPL) so that commercial use for JDK 11 and beyond will require an Oracle subscription. They do also have a non-proprietary (GPLv2+CPE) distribution but the support policy around that is unclear at this time. This is a big deal because it means that users need to have strategies and plans in place to continue to get the support that Oracle had offered. You may be wondering whether it really is critical to run OpenJDK with support and if you can get away with running it without the support. The truth is that while the open source license and nature of OpenJDK means that you can technically run the software with no commercial support, it doesn’t mean that you necessarily should. There are too many risks associated with running OpenJDK unsupported, including numerous cases of critical vulnerabilities and security flaws. While there is nothing inherently insecure about open source software, when it is deployed without support or even a maintenance plan, it can open up your organization to threats that you may not be prepared to handle and resolve. Additionally, and probably the biggest reason why it is important to have commercial OpenJDK support, is that without a third party to help, you have to be entirely self-sufficient, meaning that you would have to ensure that you have specific engineering efforts that are permanently allocated to monitoring the upstream project and maintaining installations of OpenJDK. Not all organizations have the bandwidth or resources to be able to do this consistently. How can vendors help and what are other key technology considerations? Software vendors, including Red Hat, offer OpenJDK distributions and support to make sure that those who had been reliant on Oracle JDK have a seamless transition to take care of the above risks associated with not having OpenJDK support. It also allows customers and partners to focus on their business software, rather than needing to spend valuable engineering efforts on underlying Java runtimes. Additionally, Java SE 11 has introduced some significant new features, as well as deprecated others, and is the first significant update to the platform that will require users to think seriously about the impact of moving from it. With this, it is important to separate upgrading from Java SE 8 to Java SE 11 from moving from one vendor’s Java SE distribution to another vendor’s. In fact, moving from Java SE distributions - ones that are based in OpenJDK - without needing to change versions should be a fairly simple task. It is not recommended to change both the vendor and the version in one single step. Luckily also, the differences between Oracle JDK and OpenJDK are fairly minor and in fact, are slowly aligning to become more similar. Technology vendors can help in the transition that will inevitably come for OpenJDK - if it has not happened already. Having a proper regression test plan in place to ensure that applications run as they previously did, with a particular focus on performance and scalability is key and is something that vendors can help set up. Auditing and understanding if you need to make any code changes to applications is also a major undertaking that a third-party can help guide you on, that likely includes rulesets for the migrations. Finally, third-party vendors can help you deploy to the new OpenJDK solution and provide patches and security guidance as it comes up, to help keep the environment secure, updated and safe. While there are changes to Oracle JDK, once you find the alternative solution that is best suited for your organization, it should be fairly straightforward and cost-effective to make the transition, and third party vendors have the expertise, knowledge, and experience to help guide users through the OpenJDK transition. OpenJDK Project Valhalla is now in Phase III State of OpenJDK: Past, Present and Future with Oracle Mark Reinhold on the evolution of Java platform and OpenJDK

In this tutorial, we will understand how to leverage chatbots to assist in network operations. As we move toward intelligent operations, another area to focus on is mobility. It's good to have a script to perform configurations, remediations, or even troubleshooting, but it still requires a presence to monitor, initiate, or even execute those programs or scripts. Nokia's MIKA is a good example of a chatbot that operations personnel can use for network troubleshooting and repair. According to Nokia's blog,  MIKA responds with an alarm prioritization information based on the realities for this individual network and also compare's the current situation to a whole service history of past events from this network and others, in order to identify the best solution for the current problem. Let's create a chatbot to assist in network operations. For this use case, we will use a widely-used chat application, Slack. Referring to the intelligent data analysis capabilities of Splunk, we would see some user chat interaction with the chatbot, to get some insight into the environment. This tutorial is an excerpt from a book written by Abhishek Ratan titled Practical Network Automation - Second Edition. This book will acquaint you with the fundamental concepts of network automation and help you improve your data center's robustness and security. The code for this tutorial can be found on GitHub. As we have our web framework deployed, we'll leverage the same framework to interact with the Slack chatbot, which in turn will interact with Splunk. It can also interact directly with network devices so we can initiate some complex chats, such as rebooting a router from Slack if need be. This eventually gives mobility to an engineer who can work on tasks from anywhere (even from a cellphone) without being tied to a certain location or office. To create a chatbot, here are the basic steps: Create a workspace (or account) on Slack: Create an application in your workspace (in our case, we have created an app called mybot): Here is the basic information about the application (App ID and Client ID can be used along with other information that uniquely identifies this application): Add a bot capability to this application: Add the event subscriptions and mapping to the external API that the messages will be posted to. An event subscription is when someone types the reference to the chatbot on the chat, then which API will be called with the data that is being typed in the chat with this chatbot: Here, a crucial step is once we type in the URL that accepts chat messages, that particular URL needs to be verified from Slack. A verification involves the API endpoint sending the same response back as a string or JSON that is being sent to that endpoint from Slack. If we receive the same response, Slack confirms that the endpoint is authentic and marks it as verified. This is a one-time process and any changes in the API URL will result in repeating this step. Here is the Python code in the Ops API framework that responds to this specific query: import falcon import json def on_get(self,req,resp): # Handles GET request resp.status=falcon.HTTP_200 # Default status resp.body=json.dumps({"Server is Up!"}) def on_post(self,req,resp): # Handles POST Request print("In post") data=req.bounded_stream.read() try: # Authenticating end point to Slack data=json.loads(data)["challenge"] # Default status resp.status=falcon.HTTP_200 # Send challenge string back as response resp.body=data except: # URL already verified resp.status=falcon.HTTP_200 resp.body="" This would validate, and if a challenge is sent from Slack, it would respond back with the same challenge value that confirms it to be the right endpoint for the Slack channel to send chat data to. Install this application (or chatbot) into any channels (this is similar to adding a user in a group chat): The core API framework code that responds to specific chat messages, performs the following actions: Acknowledges any post sent to Slack with a response of 200 in three seconds. If this is not done, Slack reports back: endpoint not reachable. Ensures any message sent from chatbot (not from any real user) is again not sent back as a reply. This can create a loop, since a message sent from a chatbot, would be treated as a new message in Slack chat and it would be sent again to URL. This would eventually make the chat unusable, causing repetitive messages on the chat. Authenticates the response with a token that would be sent back to Slack to ensure the response coming to Slack is from an authenticated source. The code is as follows: import falcon import json import requests import base64 from splunkquery import run from splunk_alexa import alexa from channel import channel_connect,set_data class Bot_BECJ82A3V(): def on_get(self,req,resp): # Handles GET request resp.status=falcon.HTTP_200 # Default status resp.body=json.dumps({"Server is Up!"}) def on_post(self,req,resp): # Handles POST Request print("In post") data=req.bounded_stream.read() try: bot_id=json.loads(data)["event"]["bot_id"] if bot_id=="BECJ82A3V": print("Ignore message from same bot") resp.status=falcon.HTTP_200 resp.body="" return except: print("Life goes on. . .") try: # Authenticating end point to Slack data=json.loads(data)["challenge"] # Default status resp.status=falcon.HTTP_200 # Send challenge string back as response resp.body=data except: # URL already verified resp.status=falcon.HTTP_200 resp.body="" print(data) data=json.loads(data) #Get the channel and data information channel=data["event"]["channel"] text=data["event"]["text"] # Authenticate Agent to access Slack endpoint token="xoxp-xxxxxx" # Set parameters print(type(data)) print(text) set_data(channel,token,resp) # Process request and connect to slack channel channel_connect(text) return # falcon.API instance , callable from gunicorn app= falcon.API() # instantiate helloWorld class Bot3V=Bot_BECJ82A3V() # map URL to helloWorld class app.add_route("/slack",Bot3V) Performing a channel interaction response: This code takes care of interpreting specific chats that are performed with chat-bot, in the chat channel. Additionally, this would respond with the reply, to the specific user or channel ID and with authentication token to the Slack API https://slack.com/api/chat.postMessage. This ensures the message or reply back to the Slack chat is shown on the specific channel, from where it originated. As a sample, we would use the chat to encrypt or decrypt a specific value. For example, if we write encrypt username[:]password, it would return an encrypted string with a base64 value. Similarly, if we write decrypt <encoded string>, the chatbot would return a <username/password> after decrypting the encoded string. The code is as follows: import json import requests import base64 from splunk_alexa import alexa channl="" token="" resp="" def set_data(Channel,Token,Response): global channl,token,resp channl=Channel token=Token resp=Response def send_data(text): global channl,token,res print(channl) resp = requests.post("https://slack.com/api/chat.postMessage",data='{"channel":"'+channl+'","text":"'+text+'"}',headers={"Content-type": "application/json","Authorization": "Bearer "+token},verify=False) def channel_connect(text): global channl,token,resp try: print(text) arg=text.split(' ') print(str(arg)) path=arg[0].lower() print(path in ["decode","encode"]) if path in ["decode","encode"]: print("deecode api") else: result=alexa(arg,resp) text="" try: for i in result: print(i) print(str(i.values())) for j in i.values(): print(j) text=text+' '+j #print(j) if text=="" or text==None: text="None" send_data(text) return except: text="None" send_data(text) return decode=arg[1] except: print("Please enter a string to decode") text="<decode> argument cannot be empty" send_data(text) return deencode(arg,text) def deencode(arg,text): global channl,token,resp decode=arg[1] if arg[1]=='--help': #print("Sinput") text="encode/decode <encoded_string>" send_data(text) return if arg[0].lower()=="encode": encoded=base64.b64encode(str.encode(decode)) if '[:]' in decode: text="Encoded string: "+encoded.decode('utf-8') send_data(text) return else: text="sample string format username[:]password" send_data(text) return try: creds=base64.b64decode(decode) creds=creds.decode("utf-8") except: print("problem while decoding String") text="Error decoding the string. Check your encoded string." send_data(text) return if '[:]' in str(creds): print("[:] substring exists in the decoded base64 credentials") # split based on the first match of "[:]" credentials = str(creds).split('[:]',1) username = str(credentials[0]) password = str(credentials[1]) status = 'success' else: text="encoded string is not in standard format, use username[:]password" send_data(text) print("the encoded base64 is not in standard format username[:]password") username = "Invalid" password = "Invalid" status = 'failed' temp_dict = {} temp_dict['output'] = {'username':username,'password':password} temp_dict['status'] = status temp_dict['identifier'] = "" temp_dict['type'] = "" #result.append(temp_dict) print(temp_dict) text="<username> "+username+" <password> "+password send_data(text) print(resp.text) print(resp.status_code) return This code queries the Splunk instance for a particular chat with the chatbot. The chat would ask for any management interface (Loopback45) that is currently down. Additionally, in the chat, a user can ask for all routers on which the management interface is up. This English response is converted into a Splunk query and, based upon the response from Splunk, it returns the status to the Slack chat. Let us see the code that performs the action to respond the result, to Slack chat: from splunkquery import run def alexa(data,resp): try: string=data.split(' ') except: string=data search=' '.join(string[0:-1]) param=string[-1] print("param"+param) match_dict={0:"routers management interface",1:"routers management loopback"} for no in range(2): print(match_dict[no].split(' ')) print(search.split(' ')) test=list(map(lambda x:x in search.split(' '),match_dict[no].split(' '))) print(test) print(no) if False in test: pass else: if no in [0,1]: if param.lower()=="up": query="search%20index%3D%22main%22%20earliest%3D0%20%7C%20dedup%20interface_name%2Crouter_name%20%7C%20where%20interface_name%3D%22Loopback45%22%20%20and%20interface_status%3D%22up%22%20%7C%20table%20router_name" elif param.lower()=="down": query="search%20index%3D%22main%22%20earliest%3D0%20%7C%20dedup%20interface_name%2Crouter_name%20%7C%20where%20interface_name%3D%22Loopback45%22%20%20and%20interface_status%21%3D%22up%22%20%7C%20table%20router_name" else: return "None" result=run(query,resp) return result The following Splunk query fetches the status: For UP interface: The query would be as follows: index="main" earliest=0 | dedup interface_name,router_name | where interface_name="Loopback45" and interface_status="up" | table router_name For DOWN interface (any status except ): The query would be as follows: index="main" earliest=0 | dedup interface_name,router_name | where interface_name="Loopback45" and interface_status!="up" | table router_name Let's see the end result of chatting with the chatbot and the responses being sent back based on the chats. The encoding/decoding example is as follows: As we can see here, we sent a chat with the encode abhishek[:]password123 message. This chat was sent as a POST request to the API, which in turn encrypted it to base64 and responded back with the added words as Encoded string: <encoded string>. In the next chat, we passed the same string with the decode option. This responds back with decoding the information from API function, and responds back to Slack chat, with username abhishek and password password123. Let's see the example of the Splunk query chat: In this query, we have shut down the Loopback45 interface on rtr1. During our scheduled discovery of those interfaces through the Python script, the data is now in Splunk. When queried on which management interface (Loopback45) is down, it would respond back with rtr1. The slack chat, On which routers the management interface is down, would pass this to the API, which, upon receiving this payload, will run the Splunk query to get the stats. The return value (which, in this case, is rtr1) will be given back as a response in the chat. Similarly, a reverse query of, On which routers the management interface is up, will query Splunk and eventually share back the response as rtr2, rtr3, and rtr4 (as interfaces on all these routers are UP). This chat use case can be extended to ensure that full end-to-end troubleshooting can occur using a simple chat. Extensive cases can be built using various backend functions, starting from a basic identification of problems to complex tasks, such as remediation based upon identified situations. Summary In this tutorial, we implemented some real-life use cases and looked at techniques to perform troubleshooting using chatbot. The use cases gave us insight into performing intelligent remediation as well as performing audits at scale, which are key challenges in the current environment. To learn how to automate your own network without any hassle while leveraging the power of Python, check out our book Practical Network Automation - Second Edition.  Preparing and automating a task in Python [Tutorial] PyPy 7.0 released for Python 2.7, 3.5, and 3.6 alpha 5 blog posts that could make you a better Python programmer

If you want to learn how to build artificial intelligence systems, the first step is simple: forget all about artificial intelligence. Instead focus your attention on machine learning. That way, you can be sure you’re in the domain of the practical rather than the domain of hype. Okay, this position might sound a little too dramatic. But there are a number of jokes doing the rounds on Twitter along these lines. Mat Velloso, an adviser to Satya Nadella at Microsoft, wrote late last year that “if it’s written in Python, it’s machine learning. If it’s written in PowerPoint, it’s probably AI.” https://twitter.com/matvelloso/status/1065778379612282885 There are similar jokes that focus on the use of the different words depending on whether you’re talking to investors or colleagues - either way, it’s clear that if you’re starting to explore artificial intelligence and machine learning, understanding what’s important and what you can ignore will help you to get a better view on where you need to go as your learning journey unfolds. So, once you understand that artificial intelligence is merely the word describing the end goal we’re trying to achieve, and machine learning is a means of achieving that goal, you can begin to start trying to develop intelligent systems yourself. Clearly, a question will keep cropping up: where next? Well, this post should go some way to helping you. Do you want to learn artificial intelligence? Read Packt's extensive Learning Path Python: Beginner's Guide to Artificial Intelligence. For a more advanced guide, check out Python: Advanced Guide to Artificial Intelligence. The basics of machine learning If you want to build artificial intelligence, you need to start by learning the basics of machine learning. Follow these steps: Get to grips with the basics of Python and core programming principles - if you’re reading this, you probably know enough to begin, but if you don’t there are plenty of resources to get you started. (We suggest you start with Learning Python) Make sure you understand basic statistical principles - machine learning is really just statistics, automated by code. Venturing further into machine learning and artificial intelligence The next step builds on those foundations. This is where you begin thinking about the sorts of problems you want to solve and the types of questions you want to ask. This is actually a creative step where you set the focus for your project - whatever kind of pattern or relationship you want to understand, this is where you can do just that. One of the difficulties, however, is making sure you have access to the data you need to actually do what you want. Sometimes, you might need to do some serious web scraping or data mining to get hold of the data you want - that’s beyond the scope of this piece, but there are plenty of resources out there to help you do just that. But there are also plenty of ready made data sets available for you to use in your machine learning project in whichever way you wish. You can find 50 data sets for machine learning here, all for a range of different uses. (If you’re trying machine learning for the first time, we’d suggest using one of these data sets and playing around to save you collecting data). Getting to grips with data modelling Although machine learning modelling is the next step in the learning journey, arguably it should happen at the same time as you’re thinking about both the questions you’re asking and the different data sources you might require. This is because the model - or models - you decide to employ in your project will follow directly from the problems you’re trying to tackle and, indeed, the nature and size of the data sets you eventually use. It’s important to note that no model is perfect. There’s a rule in the data science and machine learning world called the ‘no free lunch’ rule - basically, there’s no model that offers a short cut. There will always be trade offs between different algorithms in how they perform in various factors. To manage this issue you need to understand what’s important to you - maybe you’re not worried about speed, for example? Or perhaps accuracy isn’t crucial, you just want to build something that runs quickly. Broadly, the models you use will fall into these categories: supervised or unsupervised. Supervised machine learning algorithms Supervised learning is where you have an input and an output and you use an algorithm to better understand the relationship between the two. Ultimately, you want to get to a point when your machine learning system understands the relationship in such a way that you could predict an output. Supervised learning can also be broken down into regression or classification. Regression is where the output is a number or value, while classification is a specific category, or descriptor. Some algorithms can be used for both regression and classification problems, such as random forest, while others can be used for one or the other. For example, support vector machines can be used for classification problems, while linear regression algorithms can, as the name indicates, be used for regression problems. Unsupervised machine learning algorithms Unsupervised machine learning contrasts from supervised machine learning in that there are no outputs on which the algorithm works. If supervised learning 'tells' the algorithm the answers from which it then needs to understand how those answers were generated, unsupervised learning aims to understand the underlying structure within a given set of data. There aren’t any answers to guide the machine learning algorithm. As above, there are a couple of different approaches to unsupervised machine learning: clustering and association. Clustering helps you understand different groups within a set of data, while association is simply a way of understanding relationship or rules: if this happens, then this will happen too. Okay, so what about artificial intelligence? By now you will have a solid foundation of knowledge in machine learning. However, this is only the tip of the iceberg - machine learning at its most basic provides a very limited form of artificial intelligence. Advances in artificial intelligence are possible through ever more powerful algorithms - artificial or deep neural networks - that have additional layers of complexity (quite literally additional neurons). These are the algorithms that are used to power sophisticated applications and tools. From image recognition to image identification, through to speech to text and machine translation, the applications of these algorithms are radically transforming our relationship with technology. But you probably already knew that. The important question is how you actually go about doing it. Well, luckily in many ways, if you know the core components of machine learning, more advanced elements of deep learning and artificial neural networks shouldn’t actually be as complex as you might at first think. There are, however, a couple of considerations that become more important as you move deeper into deep learning. Hardware considerations for deep learning One of the most important considerations for any deep learning projects you want to try is the hardware you’re using. For a basic machine learning problem, this shouldn’t be an issue. However, but as the computations on which your deep learning system is working become more extensive, the hardware you use to run will become a challenge you need to resolve. This is too big an issue to explore here, but you can look in detail at our comparison of different processors here. Getting started with deep learning frameworks One of the reasons the whole world is talking about artificial intelligence is because it’s easier to do. And this is thanks, in part, to the growth of new deep learning frameworks that make it relatively straightforward to build complex deep learning models. The likes of TensorFlow, Keras, and PyTorch are all helping engineers and data scientists build deep learning models of considerable sophistication. Although they each have their own advantages, and it’s well worth spending some time comparing them, there’s certainly a lot to be said for simply getting started with them yourself. What about cloud's impact on machine learning and artificial intelligence? An interesting development in the machine learning space is the impact of cloud based solutions. The likes of Azure, AWS and Google Cloud Platform are all offering a number of different services and tools from within their overarching cloud products that make performing machine and deep learning tasks much easier. While this is undoubtedly going to be an important development, and, indeed, one you may have encountered already, there is no substitute for simply getting your hands dirty with the data and seeing how the core principles behind machine learning and artificial intelligence actually work. Conclusion: Don’t be scared, take machine learning and artificial intelligence one step at a time Clearly, with so much hype around artificial intelligence its easy to get stuck before you begin. However, by focusing on the core principles and practical application of machine learning you will be well on your way to helping drive the future of artificial intelligence. Learn artificial intelligence from scratch with Python: Beginner's Guide to Artificial Intelligence. Dive deeper into deep learning and artificial intelligence with Python: Advanced Guide to Artificial Intelligence.  

To implement a use-case, we need to use a well-known Spring module, Spring Web and Spring Web MVC. Our application will not use the new features of Spring 5, so it will run similarly on Spring Framework 4.x. This article is an excerpt taken from the book Hands-On Reactive Programming in Spring 5 by Oleh Dokuka and Igor Lozynskyi. This book covers the difference between a reactive system and reactive programming, the basics of reactive programming in Spring 5 and much more. In this article, you will learn how to bootstrap a Spring application, implement business logic, and much more. To bootstrap our application, we may configure and download a Gradle project from the Spring Initializer website at start.spring.io. For now, we need to select the preferred Spring Boot version and dependency for the web (the actual dependency identifier in Gradle config will be org.springframework.boot:spring-boot-starter-web), as shown in the following screenshot: Diagram 2.4 Web-based Spring Initializer simplifies the bootstrapping of a new Spring Boot application Alternatively, we may generate a new Spring Boot project using cURL and the HTTP API of the Spring Boot Initializer site. The following command will effectively create and download the same empty project with all the desired dependencies: curl https://start.spring.io/starter.zip \ -d dependencies=web,actuator \ -d type=gradle-project \ -d bootVersion=2.0.2.RELEASE \ -d groupId=com.example.rpws.chapters \ -d artifactId=SpringBootAwesome \ -o SpringBootAwesome.zip Implementing business logic We may now outline the design of our system in the following diagram: Diagram 2.5 Events flow from a temperature sensor to a user In this use case, the domain model will consist only of the Temperature class with the only double value inside. For simplicity purposes, it is also used as an event object, as shown in the following code: final class Temperature { private final double value; // constructor & getter... } To simulate the sensor, let's implement the TemperatureSensor class and decorate it with a @Component annotation to register the Spring bean, as follows: @Component public class TemperatureSensor { private final ApplicationEventPublisher publisher; // (1) private final Random rnd = new Random(); // (2) private final ScheduledExecutorService executor = // (3) Executors.newSingleThreadScheduledExecutor(); public TemperatureSensor(ApplicationEventPublisher publisher) { this.publisher = publisher; } @PostConstruct public void startProcessing() { // (4) this.executor.schedule(this::probe, 1, SECONDS); } private void probe() { // (5) double temperature = 16 + rnd.nextGaussian() * 10; publisher.publishEvent(new Temperature(temperature)); // schedule the next read after some random delay (0-5 seconds) executor .schedule(this::probe, rnd.nextInt(5000), MILLISECONDS); // (5.1) } } So, our simulated temperature sensor only depends on the ApplicationEventPublisher class (1), provided by Spring Framework. This class makes it possible to publish events to the system. It is a requirement to have a random generator (2) to contrive temperatures with some random intervals. An event generation process happens in a separate ScheduledExecutorService (3), where each event's generation schedules the next round of an event's generation with a random delay (5.1). All that logic is defined in the probe() method (5).  In turn, the mentioned class has the startProcessing() method annotated with @PostConstruct (4), which is called by Spring Framework when the bean is ready and triggers the whole sequence of random temperature values. Asynchronous HTTP with Spring Web MVC The introduced in Servlet 3.0 asynchronous support expands the ability to process an HTTP request in non-container threads. Such a feature is pretty useful for long-running tasks. With those changes, in Spring Web MVC we can return not only a value of type T in @Controller but also a Callable<T> or a DeferredResult<T>. The Callable<T> may be run inside a non-container thread, but still, it would be a blocking call. In contrast, DeferredResult<T> allows an asynchronous response generation on a non-container thread by calling the setResult(T result) method so it could be used within the event-loop. Starting from version 4.2, Spring Web MVC makes it possible to return ResponseBodyEmitter, which behaves similarly to DeferredResult, but can be used to send multiple objects, where each object is written separately with an instance of a message converter (defined by the HttpMessageConverter interface). The SseEmitter extends ResponseBodyEmitter and makes it possible to send many outgoing messages for one incoming request in accordance with SSE's protocol requirements. Alongside ResponseBodyEmitter and SseEmitter, Spring Web MVC also respects the StreamingResponseBody interface. When returned from @Controller, it allows us to send raw data (payload bytes) asynchronously. StreamingResponseBody may be very handy for streaming large files without blocking Servlet threads. Exposing the SSE (Server Sent Events) endpoint The next step requires adding the TemperatureController class with the @RestController annotation, which means that the component is used for HTTP communication, as shown in the following code: @RestController public class TemperatureController { private final Set<SseEmitter> clients = // (1) new CopyOnWriteArraySet<>(); @RequestMapping( value = "/temperature-stream", // (2) method = RequestMethod.GET) public SseEmitter events(HttpServletRequest request) { // (3) SseEmitter emitter = new SseEmitter(); // (4) clients.add(emitter); // (5) // Remove emitter from clients on error or disconnect emitter.onTimeout(() -> clients.remove(emitter)); // (6) emitter.onCompletion(() -> clients.remove(emitter)); // (7) return emitter; // (8) } @Async // (9) @EventListener // (10) public void handleMessage(Temperature temperature) { // (11) List<SseEmitter> deadEmitters = new ArrayList<>(); // (12) clients.forEach(emitter -> { try { emitter.send(temperature, MediaType.APPLICATION_JSON); // (13) } catch (Exception ignore) { deadEmitters.add(emitter); // (14) } }); clients.removeAll(deadEmitters); // (15) } } Now, to understand the logic of the TemperatureController class, we need to describe the SseEmitter. Spring Web MVC provides that class with the sole purpose of sending SSE events. When a request-handling method returns the SseEmitter instance, the actual request processing continues until SseEnitter.complete(), an error, or a timeout occurs. The TemperatureController provides one request handler (3) for the URI /temperature-stream (2) and returns the SseEmitter (8). In the case when a client requests that URI, we create and return the new SseEmitter instance (4) with its previous registration in the list of the active clients (5). Furthermore, the SseEmitter constructor may consume the timeout parameter. For the clients' collection, we may use the CopyOnWriteArraySet class from the java.util.concurrent package (1). Such an implementation allows us to modify the list and iterate over it at the same time. When a web client opens a new SSE session, we add a new emitter to the clients' collection. The SseEmitter removes itself from the clients' list when it has finished processing or has reached timeout (6) (7). Now, having a communication channel with clients means that we need to be able to receive events about temperature changes. For that purpose, our class has a handleMessage() method (11). It is decorated with the @EventListener annotation (10) in order to receive events from Spring. This framework will invoke the handleMessage() method only when receiving Temperature events, as this type of method's argument is known as temperature. The @Async annotation (9) marks a method as a candidate for the asynchronous execution, so it is invoked in the manually configured thread pool. The handleMessage() method receives a new temperature event and asynchronously sends it to all clients in JSON format in parallel for each event (13). Also, when sending to individual emitters, we track all failing ones (14) and remove them from the list of the active clients (15). Such an approach makes it possible to spot clients that are not operational anymore. Unfortunately, SseEmitter does not provide any callback for handling errors, and can be done by handling errors thrown by the send() method only. Configuring asynchronous support To run everything, we need an entry point for our application with the following customized methods: @EnableAsync // (1) @SpringBootApplication // (2) public class Application implements AsyncConfigurer { public static void main(String[] args) { SpringApplication.run(Application.class, args); } @Override public Executor getAsyncExecutor() { // (3) ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();// (4) executor.setCorePoolSize(2); executor.setMaxPoolSize(100); executor.setQueueCapacity(5); // (5) executor.initialize(); return executor; } @Override public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler(){ return new SimpleAsyncUncaughtExceptionHandler(); // (6) } } As we can see, the example is a Spring Boot application (2), with an asynchronous execution enabled by the @EnableAsync annotation (1). Here, we may configure an exception handler for exceptions thrown from the asynchronous execution (6). That is also where we prepare Executor for asynchronous processing. In our case, we use ThreadPoolTaskExecutor with two core threads that may be increased to up to one hundred threads. It is important to note that without a properly configured queue capacity (5), the thread pool is not able to grow. That is because the SynchronousQueue would be used instead, limiting concurrency. Building a UI with SSE support The last thing that we need in order to complete our use case is an HTML page with some JavaScript code to communicate with the server. For the sake of conciseness, we will strip all HTML tags and leave only the minimum that is required to achieve a result, as follows: <body> <ul id="events"></ul> <script type="application/javascript"> function add(message) { const el = document.createElement("li"); el.innerHTML = message; document.getElementById("events").appendChild(el); } var eventSource = new EventSource("/temperature-stream"); // (1) eventSource.onmessage = e => { // (2) const t = JSON.parse(e.data); const fixed = Number(t.value).toFixed(2); add('Temperature: ' + fixed + ' C'); } eventSource.onopen = e => add('Connection opened'); // (3) eventSource.onerror = e => add('Connection closed'); // </script> </body> Here, we are using the EventSource object pointed at /temperature-stream (1). This handles incoming messages by invoking the onmessage() function (2), error handling, and reaction to the stream opening, which are done in the same fashion (3). We should save this page as index.html and put it in the src/main/resources/static/ folder of our project. By default, Spring Web MVC serves the content of the folder through HTTP. Such behavior could be changed by providing a configuration that extends the WebMvcConfigurerAdapter class. Verifying application functionality After rebuilding and completing our application's startup, we should be able to access the mentioned web page in a browser at the following address: http://localhost:8080 (Spring Web MVC uses port 8080 for the web server as the default one. However, this can be changed in the application.properties file using the configuration line server.port=9090). After a few seconds, we may see the following output: Connection opened Temperature: 14.71 C Temperature: 9.67 C Temperature: 19.02 C Connection closed Connection opened Temperature: 18.01 C Temperature: 16.17 C As we can see, our web page reactively receives events, preserving both client and server resources. It also supports auto-reconnect in the case of network issues or timeouts. As the current solution is not exclusive to JavaScript, we may connect with other clients for example, curl. By running the next command in a terminal, we receive the following stream of raw, but not formatted, events: > curl http://localhost:8080/temperature-stream data:{"value":22.33210856124129} data:{"value":13.83133638119636} In this article, we learned how to bootstrap a Spring application, implement business logic, and much more. To know more about the difference between a reactive system and reactive programming, check out the book Hands-On Reactive Programming in Spring 5 by Oleh Dokuka and Igor Lozynskyi. Netflix adopts Spring Boot as its core Java framework Implementing Dependency Injection in Spring [Tutorial] How to recover deleted data from an Android device [Tutorial]

Last Saturday, Google presented a paper in the Munich Security Conference titled How Google Fights Disinformation. In the paper, they explain what steps they’re taking against disinformation and detail their strategy for their platforms Google Search, News, YouTube, and Google Ads. We take a look at the key strategies that Google is taking against disinformation. Disinformation has become widespread in recent years. It directly affects Google’s mission of organizing the world’s information and making it accessible. Disinformation, misinformation, or fake new are deliberate attempts by acting parties to mislead people in believing things that aren’t true by spreading such content over the internet. Disinformation is deliberate attempts to mislead people where the creator knows that the information is false, misinformation is where the creator has their facts wrong and spreads wrong information unintentionally. The motivations behind it can be financial, political, or just for entertainment (trolls). Motivations can overlap with the content produced, moreover, the disinformation could also be for a good cause, making the fight against fake news very complex. A common solution for all platforms is not possible as different platforms pose different challenges. Making standards that exercise deep deliberation for individual cases is also not practical. There are three main principles that Google is outlining to combat disinformation, shown as follows. #1 Make quality content count Google products sort through a lot of information to display the most useful content first. They want to deliver quality content and legitimate commercial messages are prone to rumors. While the content is different on different Google platforms, the principles are similar: Organizing information by ranking algorithms. The algorithms are aimed to ensure that the information benefits users and is measured by user testing #2 Counter malicious actors Algorithms cannot determine if a piece of content is true or false based on current events. Neither can it determine the true intents of the content creator. For this, Google products have policies that prohibit certain behaviors like misinterpreting ownership of content. Certain users try to get a better ranking by practicing spam, such behavior is also shown by people who engage in spreading disinformation. Google has algorithms in place that can reduce such content and it’ll also be supported by human reviews for further filtering. #3 Giving users more choices Giving users different perspectives is important before they choose a link and proceed reading content or viewing a video. Hence, Google provides multiple links for a topic searched. Google search and other products now have additional UI elements to segregate information into different sections for an organized view of content. They also have a feedback button on their services via which users can submit their thoughts. Partnership with external experts Google cannot do this alone, hence they have partnered with supporting new organizations to create quality content that can uproot disinformation. They mention in the paper: “In March 2018, we launched the Google News Initiative (GNI) 3 to help journalism thrive in the digital age. With a $300 million commitment over 3 years, the initiative aims to elevate and strengthen quality journalism.” Preparing for the future People who create fake news will always try new methods to propagate it. Google is investing in research and development against it, now especially before the elections. They intend to stay ahead of the malicious actors who may use new technologies or tactics which can include deepfakes. They want to protect so that polling booths etc are easily available, guard against phishing, mitigate DDoS attacks on political websites. YouTube and conspiracy theories Recently, there have been a lot of conspiracy theories floating around on YouTube. In the paper, they say that: “YouTube has been developing products that directly address a core vulnerability involving the spread of disinformation in the immediate aftermath of a breaking news event.” Making a legitimate video with correct facts takes time, while disinformation can be created quickly for spreading panic/negativity etc,. In conclusion they, note that “fighting disinformation is not a straightforward endeavor. Disinformation and misinformation can take many shapes, manifest differently in different products, and raise significant challenges when it comes to balancing risks of harm to good faith, free expression, with the imperative to serve users with information they can trust.” Public reactions People think that only the platforms themselves can take actions against disinformation propaganda. https://twitter.com/halhod/status/1097640819102691328 Users question Google’s efforts in cases where the legitimate website is shown after the one with disinformation with an example of Bitcoin. https://twitter.com/PilotDaveCrypto/status/1097395466734653440 Some speculate that corporate companies should address their own bias of ranking pages first: https://twitter.com/PaulJayzilla/status/1097822412815646721 https://twitter.com/Darin_T80/status/1097203275483426816 To read the complete research paper with Google product-specific details on fighting disinformation, you can head on to the Google Blog. Fake news is a danger to democracy. These researchers are using deep learning to model fake news to understand its impact on elections. Defending Democracy Program: How Microsoft is taking steps to curb increasing cybersecurity threats to democracy Is Anti-trust regulation coming to Facebook following fake news inquiry made by a global panel in the House of Commons, UK?

When dealing with security, it makes sense to keep those levels in mind in order to approach security-related issues in an organized way. Bind addresses: listen_addresses in the postgresql.conf file Host-based access control: The pg_hba.conf file Instance-level permissions: Users, roles, database creation, login, and replication Database-level permissions: Connecting, creating schemas, and so on Schema-level permissions: Using schema and creating objects inside a schema Table-level permissions: Selecting, inserting, updating, and so on Column-level permissions: Allowing or restricting access to columns Row-level security: Restricting access to rows In order to read a value, PostgreSQL has to ensure that we have sufficient permissions on every level. The entire chain of permissions has to be correct. This article is an excerpt taken from the book Mastering PostgreSQL 11 - Second Edition by Hans-Jürgen Schönig. In this book, you will learn the approach to get to grips with advanced PostgreSQL 11 features and SQL functions, master replication and failover techniques, configure database security and more. In this article, you will learn the process of handling SSL, column-level security and configuring default privileges and much more. Understanding bind addresses and connections When configuring a PostgreSQL server, one of the first things that needs to be done is define remote access. By default, PostgreSQL does not accept remote connections. The important thing here is that PostgreSQL does not even reject the connection because it simply does not listen on the port. If we try to connect, the error message will actually come from the operating system because PostgreSQL does not care at all. Assuming that there is a database server using the default configuration on 192.168.0.123, the following will happen: iMac:~ hs$ telnet 192.168.0.123 5432 Trying 192.168.0.123... telnet: connect to address 192.168.0.123: Connection refused telnet: Unable to connect to remote host Telnet tries to create a connection on port 5432 and is instantly rejected by the remote box. From the outside, it looks as if PostgreSQL is not running at all. The key to success can be found in the postgresql.conf file: # - Connection Settings - # listen_addresses = 'localhost' # what IP address(es) to listen on; # comma-separated list of addresses; # defaults to 'localhost'; use '*' for all # (change requires restart) The listen_addresses setting will tell PostgreSQL which addresses to listen on. Technically speaking, those addresses are bind addresses. What does that actually mean? Suppose we have four network cards in our machine. We can listen on, say, three of those IP addresses. PostgreSQL takes requests to those three cards into account and does not listen on the fourth one. The port is simply closed. If we put an * in, PostgreSQL will listen to every IP assigned to your machine. However, there are more settings related to connection management that are highly important to understand. They are as follows: #port = 5432 # (change requires restart) max_connections = 100 # (change requires restart) # Note: Increasing max_connections costs ~400 bytes of # shared memory per # connection slot, plus lock space # (see max_locks_per_transaction). #superuser_reserved_connections = 3 # (change requires restart) #unix_socket_directories = '/tmp' # comma-separated list of directories # (change requires restart) #unix_socket_group = '' # (change requires restart) #unix_socket_permissions = 0777 # begin with 0 to use octal notation # (change requires restart) First of all, PostgreSQL listens to a single TCP port, the default value of which is is 5432. Keep in mind that PostgreSQL will listen on a single port only. Whenever a request comes in, the postmaster will fork and create a new process to handle the connection. By default, up to 100 normal connections are allowed. On top of that, three additional connections are reserved for superusers. This means that we can either have 97 connections plus three superusers or 100 superuser connections. Handling SSL PostgreSQL allows us to encrypt the transfer between the server and the client. Encryption is highly beneficial, especially if we are communicating over long distances. SSL offers a simple and secure way to ensure that nobody is able to listen to your communication. In this section, we will learn how to set up SSL. The first thing to do is to set the ssl parameter to on in the postgresql.conf file when the server starts. In the next step, we can put SSL certificates into the $PGDATA directory. If we don't want the certificates to be in some other directory, change the following parameters: #ssl_cert_file = 'server.crt' # (change requires restart) #ssl_key_file = 'server.key' # (change requires restart) #ssl_ca_file = '' # (change requires restart) #ssl_crl_file = '' # (change requires restart) If we want to use self-signed certificates, perform the following step: openssl req -new -text -out server.req Answer the questions asked by OpenSSL. Make sure that we enter the local hostname as common name. We can leave the password empty. This call will generate a key that is passphrase protected; it will not accept a passphrase that is less than four characters long. To remove the passphrase (as you must if you want automatic startup of the server), run the following commands: openssl rsa -in privkey.pem -out server.key rm privkey.pem Enter the old passphrase to unlock the existing key. Now, do this to turn the certificate into a self-signed certificate and to copy the key and certificate to where the server will look for them: openssl req -x509 -in server.req -text -key server.key -out server.crt After doing this, make sure that the files have the right set of permissions: chmod og-rwx server.key Once the proper rules have been put into the pg_hba.conf file, we can use SSL to connect to your server. To verify that we are indeed using SSL, consider checking out the pg_stat_ssl function. It will tell us every connection and whether it uses SSL or not, and it will provide some important information about encryption: test=# \d pg_stat_sslView "pg_catalog.pg_stat_ssl" Column | Type | Modifiers -------------+----------+----------- pid | integer | ssl | boolean | version | text | cipher | text | bits | integer | compression | boolean | clientdn | text | If the ssl field for a process contains true; PostgreSQL does what we would expect it to do: postgres=# select * from pg_stat_ssl; -[ RECORD 1 ] ---------------------------- pid | 20075 ssl | t version | TLSv1.2 cipher | ECDHE-RSA-AES256-GCM-SHA384 bits | 256 compression | f clientdn | Handling instance-level security So far, we have configured bind addresses and we have told PostgreSQL which means of authentication to use for which IP ranges. Up to now, the configuration was purely network-related. In the next step, we can shift our attention to permissions at the instance level. The most important thing to know is that users in PostgreSQL exist at the instance level. If we create a user, it is not just visible inside one database; it can be seen by all the databases. A user might have permissions to access just a single database, but basically users are created at the instance level. To those of you who are new to PostgreSQL, there is one more thing you should keep in mind: users and roles are the same thing. CREATE ROLE and CREATE USER clauses have different default values (literally, the only difference is that roles do not get the LOGIN attribute by default), but at the end of the day, users and roles are the same. Therefore, CREATE ROLE and CREATE USER clauses support the very same syntax: test=# \h CREATE USER Command: CREATE USER Description: define a new database role Syntax: CREATE USER name [ [ WITH ] option [ ... ] ] where option can be: SUPERUSER | NOSUPERUSER | CREATEDB | NOCREATEDB | CREATEROLE | NOCREATEROLE | INHERIT | NOINHERIT | LOGIN | NOLOGIN | REPLICATION | NOREPLICATION | BYPASSRLS | NOBYPASSRLS | CONNECTION LIMIT connlimit | [ ENCRYPTED ] PASSWORD 'password' | VALID UNTIL 'timestamp' | IN ROLE role_name [, ...] | IN GROUP role_name [, ...] | ROLE role_name [, ...] | ADMIN role_name [, ...] | USER role_name [, ...] | SYSID uid Let's discuss those syntax elements one by one. The first thing we see is that a user can be a superuser or a normal user. If somebody is marked as a SUPERUSER , there are no longer any restrictions that a normal user has to face. A SUPERUSER can drop objects (databases and so on) as they wish. The next important thing is that it takes permissions on the instance level to create a new database. The rule is this: the creator is always automatically the owner of an object (unless specified otherwise, as can be done with the CREATE DATABASE clause). The beauty is that object owners can also drop an object again. The next important thing is the INHERIT/NOINHERIT clause. If the INHERIT clause is set (which is the default value), a user can inherit permissions from some other user. Using inherited permissions allows us to use roles, which is as a good way to abstract permissions. For example, we can create the role of bookkeeper and make many other roles inherit from bookkeeper. The idea is that we only have to tell PostgreSQL once what a bookkeeper is allowed to do, even if we have many people working in accounting. The LOGIN/NOLOGIN clause defines whether a role is allowed to log in to the instance. After this theoretical introduction, it is time to actually create users and see how things can be used in a practical example: test=# CREATE ROLE bookkeeper NOLOGIN; CREATE ROLE test=# CREATE ROLE joe LOGIN; CREATE ROLE test=# GRANT bookkeeper TO joe; GRANT ROLE The first thing done here is that a role called bookkeeper is created. Note that we don't want people to log in as bookkeeper, so the role is marked as NOLOGIN. Also note that NOLOGIN is the default value if you use the CREATE ROLE clause. If you prefer the CREATE USER clause, the default setting is LOGIN. Then, the joe role is created and marked as LOGIN. Finally, the bookkeeper role is assigned to the joe role so that he can do everything a bookkeeper is actually allowed to do. Once the users are in place, we can test what we have so far: [hs@zenbook ~]$ psql test -U bookkeeper psql: FATAL: role "bookkeeper" is not permitted to log in As expected, the bookkeeper role is not allowed to log in to the system. What happens if the joe role tries to log in? [hs@zenbook ~]$ psql test -U joe ... test=> This will actually work as expected. However, note that Command Prompt has changed. This is just a way for PostgreSQL to show you that you are not logged in as a superuser. Once a user has been created, it might be necessary to modify it. One thing we might want to change is the password. In PostgreSQL, users are allowed to change their own passwords. Here is how it works: test=> ALTER ROLE joe PASSWORD 'abc'; ALTER ROLE test=> SELECT current_user; current_user -------------- joe (1 row) The ALTER ROLE clause (or ALTER USER) will allow us to change most settings which can be set during user creation. However, there is even more to managing users. In many cases, we want to assign special parameters to a user. The ALTER USER clause gives us the means to do that: ALTER ROLE { role_specification | ALL } [ IN DATABASE database_name ] SET configuration_parameter { TO | = } { value | DEFAULT } ALTER ROLE { role_specification | ALL } [ IN DATABASE database_name ] SET configuration_parameter FROM CURRENT ALTER ROLE { role_specification | ALL } [ IN DATABASE database_name ] RESET configuration_parameter ALTER ROLE { role_specification | ALL } [ IN DATABASE database_name ] RESET ALL The syntax is fairly simple and pretty straightforward. To depict why this is really useful, I have added a real-world example. Let's suppose that Joe happens to live on the island of Mauritius. When he logs in, he wants to be in his own time zone, even if his database server is located in Europe: test=> ALTER ROLE joe SET TimeZone = 'UTC-4'; ALTER ROLE test=> SELECT now(); now ------------------------------- 2017-01-09 20:36:48.571584+01 (1 row) test=> q [hs@zenbook ~]$ psql test -U joe ... test=> SELECT now(); now ------------------------------- 2017-01-09 23:36:53.357845+04 (1 row) The ALTER ROLE clause will modify the user. As soon as joe reconnects, the time zone will already be set for him. The time zone is not changed immediately. You should either reconnect or use a SET ... TO DEFAULT clause. The important thing here is that this is also possible for some memory parameters, such as work_mem and so on. Security at the database level After configuring users at the instance level, it is possible to dig deeper and see what can be done at the database level. The first major question that arises is: we explicitly allowed Joe to log in to the database instance, but who or what allowed Joe to actually connect to one of the databases? Maybe we don't want Joe to access all the databases in your system. Restricting access to certain databases is exactly what we can achieve on this level. For databases, the following permissions can be set using a GRANT clause: GRANT { { CREATE | CONNECT | TEMPORARY | TEMP } [, ...] | ALL [ PRIVILEGES ] } ON DATABASE database_name [, ...] TO role_specification [, ...] [ WITH GRANT OPTION ] There are two major permissions on the database level that deserve close attention: CREATE: This allows somebody to create a schema inside the database. Note that a CREATE clause does not allow for the creation of tables; it is about schemas. In PostgreSQL, a table resides inside a schema, so you have to get to the schema level first to be able to create a table. CONNECT: This allows somebody to connect to a database. The question now is: nobody has explicitly assigned CONNECT permissions to the joe role, so where do those permissions actually come from? The answer is this: there is a thing called public, which is similar to the Unix world. If the world is allowed to do something, so is joe, who is part of the general public. The main thing is that public is not a role in the sense that it can be dropped and renamed. We can simply see it as the equivalent for everybody on the system. So, to ensure that not everybody can connect to any database at any time, CONNECT may have to be revoked from the general public. To do so, we can connect as superuser and fix the problem: [hs@zenbook ~]$ psql test -U postgres ... test=# REVOKE ALL ON DATABASE test FROM public; REVOKE test=# \q [hs@zenbook ~]$ psql test -U joe psql: FATAL: permission denied for database "test" DETAIL: User does not have CONNECT privilege. As we can see, the joe role is not allowed to connect anymore. At this point, only superusers have access to test. In general, it is a good idea to revoke permissions from the postgres database even before other databases are created. The idea behind this concept is that those permissions won't be in all those newly created databases anymore. If somebody needs access to a certain database, rights have to be explicitly granted. Rights are not automatically there anymore. If we want to allow the joe role to connect to the test database, try the following line as superuser: [hs@zenbook ~]$ psql test -U postgres ... test=# GRANT CONNECT ON DATABASE test TO bookkeeper; GRANT test=# \q [hs@zenbook ~]$ psql test -U joe ... test=> Basically, there are two choices here: We can allow the joe role directly so that only the joe role will be able to connect. Alternatively, we can grant permissions to the bookkeeper role. Remember, the joe role will inherit all the permissions from the bookkeeper role, so if we want all accountants to be able to connect to the database, assigning permissions to the bookkeeper role seems such as an attractive idea. If we grant permissions to the bookkeeper role, it is not risky because the role is not allowed to log in to the instance in the first place, so it purely serves as a source of permissions. Handling column-level security In some cases, not everybody is allowed to see all the data. Just imagine a bank. Some people might see the entire information about a bank account, while others might be limited to only a subset of the data. In a real-world situation, somebody might not be allowed to read the balance column or somebody might not see the interest rates of people's loans. Another example would be that people are allowed to see people's profiles but not their pictures or some other private information. The question now is: how can column-level security be used? To demonstrate that, we will add a column to the existing table belonging to the joe role: test=> ALTER TABLE t_useful ADD COLUMN name text; ALTER TABLE The table now consists of two columns. The goal of the example is to ensure that a user can see only one of those columns: test=> \d t_useful Table "public.t_useful" Column | Type | Modifiers --------+---------+----------- id | integer | name | text | As a superuser, let's create a user and give it access to the schema containing our table: test=# CREATE ROLE paul LOGIN; CREATE ROLE test=# GRANT CONNECT ON DATABASE test TO paul; GRANT test=# GRANT USAGE ON SCHEMA public TO paul; GRANT CONNECT was revoked from public. Explicit granting is therefore absolutely necessary to ensure that we can even get to the table. The SELECT permissions can be given to the paul role: test=# GRANT SELECT (id) ON t_useful TO paul; GRANT Basically, this is already enough. It is already possible to connect to the database as user paul and read the column: [hs@zenbook ~]$ psql test -U paul ... test=> SELECT id FROM t_useful; id ---- (0 rows) If we are using column-level permissions, there is an important thing to keep in mind; we should stop using SELECT *, as it does not work anymore: test=> SELECT * FROM t_useful; ERROR: permission denied for relation t_useful * still means all columns, but as there is no way to access all columns, things will error out instantly. Configuring default privileges So far, a lot of stuff has already been configured. What happens if new tables are added to the system? It can be quite painful and risky to process these tables one by one and to set proper permissions. Wouldn't it be nice if those things would just happen automatically? This is exactly what the ALTER DEFAULT PRIVILEGES clause does. The idea is to give users an option to make PostgreSQL automatically set the desired permissions as soon as an object comes into existence. It cannot happen anymore that somebody simply forgets to set those rights. The following listing shows the first part of the syntax specification: postgres=# \h ALTER DEFAULT PRIVILEGES Command: ALTER DEFAULT PRIVILEGES Description: define default access privileges Syntax: ALTER DEFAULT PRIVILEGES [ FOR { ROLE | USER } target_role [, ...] ] [ IN SCHEMA schema_name [, ...] ] abbreviated_grant_or_revoke where abbreviated_grant_or_revoke is one of: GRANT { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER } [, ...] | ALL [ PRIVILEGES ] } ON TABLES TO { [ GROUP ] role_name | PUBLIC } [, ...] [ WITH GRANT OPTION ] ... Basically, the syntax works similar to the GRANT clause and is therefore easy and intuitive to use. To show us how it works, I compiled a simple example. The idea is that if the joe role creates a table, the paul role will automatically be able to use it: test=# ALTER DEFAULT PRIVILEGES FOR ROLE joe IN SCHEMA public GRANT ALL ON TABLES TO paul; ALTER DEFAULT PRIVILEGES Let's connect as the joe role now and create a table: [hs@zenbook ~]$ psql test -U joe ... test=> CREATE TABLE t_user (id serial, name text, passwd text); CREATE TABLE Connecting as the paul role will prove that the table has been assigned to the proper set of permissions: [hs@zenbook ~]$ psql test -U paul ... test=> SELECT * FROM t_user; id | name | passwd ----+------+-------- (0 row In the article, we covered the process of handling SSL, column-level security and configuring default privileges and much more. To know about row-level security and get detailed information about advanced PostgreSQL 11 features, check out the book Mastering PostgreSQL 11 - Second Edition. PostgreSQL wins ‘DBMS of the year’ 2018 beating MongoDB and Redis in DB-Engines Ranking Devart releases standard edition of dbForge Studio for PostgreSQL PipelineDB 1.0.0, the high performance time-series aggregation for PostgreSQL, released!

A siamese network is a special type of neural network and it is one of the simplest and most popularly used one-shot learning algorithms. One-shot learning is a technique where we learn from only one training example per class. So, a siamese network is predominantly used in applications where we don't have many data points in each class. For instance, let's say we want to build a face recognition model for our organization and about 500 people are working in our organization. If we want to build our face recognition model using a Convolutional Neural Network (CNN) from scratch, then we need many images of all of these 500 people for training the network and attaining good accuracy. But apparently, we will not have many images for all of these 500 people and so it is not feasible to build a model using a CNN or any deep learning algorithm unless we have sufficient data points. So, in these kinds of scenarios, we can resort to a sophisticated one-shot learning algorithm such as a siamese network, which can learn from fewer data points. Siamese networks basically consist of two symmetrical neural networks both sharing the same weights and architecture and both joined together at the end using some energy function, E. The objective of our siamese network is to learn whether two input values are similar or dissimilar. We will understand the siamese network by building a face recognition model. The objective of our network is to understand whether two faces are similar or dissimilar. We use the AT&T Database of Faces, which can be downloaded from the Cambridge University Computer Laboratory website. This article is an excerpt from a book written by Sudharsan Ravichandiran titled Hands-On Meta-Learning with Python. In this book, you will learn how to build relation networks and matching networks from scratch. Once you have downloaded and extracted the archive, you can see the folders s1, s2, up to s40, as shown here:   Each of these folders has 10 different images of a single person taken from various angles. For instance, let's open folder s1. As you can see, there are 10 different images of a single person:   We open and check folder s13: Siamese networks require input values as a pair along with the label, so we have to create our data in such a way. So, we will take two images randomly from the same folder and mark them as a genuine pair and we will take single images from two different folders and mark them as an imposite pair. A sample is shown in the following screenshot; as you can see, a genuine pair has images of the same person and the imposite pair has images of different people: Once we have our data as pairs along with their labels, we train our siamese network. From the image pair, we feed one image to network A and another image to network B. The role of these two networks is only to extract the feature vectors. So, we use two convolution layers with rectified linear unit (ReLU) activations for extracting the features. Once we have learned the features, we feed the resultant feature vector from both of the networks to the energy function, which measures the similarity; we use Euclidean distance as our energy function. So, we train our network by feeding the image pair to learn the semantic similarity between them. Now, we will see this step by step. For better understanding, you can check the complete code, which is available as a Jupyter Notebook with an explanation from GitHub. First, we will import the required libraries: import re import numpy as np from PIL import Image from sklearn.model_selection import train_test_split from keras import backend as K from keras.layers import Activation from keras.layers import Input, Lambda, Dense, Dropout, Convolution2D, MaxPooling2D, Flatten from keras.models import Sequential, Model from keras.optimizers import RMSprop Now, we define a function for reading our input image. The read_image function takes as input an image and returns a NumPy array: def read_image(filename, byteorder='>'): #first we read the image, as a raw file to the buffer with open(filename, 'rb') as f: buffer = f.read() #using regex, we extract the header, width, height and maxval of the image header, width, height, maxval = re.search( b"(^P5\s(?:\s*#.*[\r\n])*" b"(\d+)\s(?:\s*#.*[\r\n])*" b"(\d+)\s(?:\s*#.*[\r\n])*" b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups() #then we convert the image to numpy array using np.frombuffer which interprets buffer as one dimensional array return np.frombuffer(buffer, dtype='u1' if int(maxval) < 256 else byteorder+'u2', count=int(width)*int(height), offset=len(header) ).reshape((int(height), int(width))) For an example, let's open one image: Image.open("data/orl_faces/s1/1.pgm") When we feed this image to our read_image function, it will return as a NumPy array: img = read_image('data/orl_faces/s1/1.pgm') img.shape (112, 92) Now, we define another function, get_data, for generating our data. As we know, for the siamese network, data should be in the form of pairs (genuine and imposite) with a binary label. First, we read the (img1, img2) images from the same directory and store them in the x_genuine_pair array and assign y_genuine to 1. Next, we read the (img1, img2) images from the different directory and store them in the x_imposite pair and assign y_imposite to 0. Finally, we concatenate both x_genuine_pair and x_imposite to X and y_genuine and y_imposite to Y: size = 2 total_sample_size = 10000 def get_data(size, total_sample_size): #read the image image = read_image('data/orl_faces/s' + str(1) + '/' + str(1) + '.pgm', 'rw+') #reduce the size image = image[::size, ::size] #get the new size dim1 = image.shape[0] dim2 = image.shape[1] count = 0 #initialize the numpy array with the shape of [total_sample, no_of_pairs, dim1, dim2] x_geuine_pair = np.zeros([total_sample_size, 2, 1, dim1, dim2]) # 2 is for pairs y_genuine = np.zeros([total_sample_size, 1]) for i in range(40): for j in range(int(total_sample_size/40)): ind1 = 0 ind2 = 0 #read images from same directory (genuine pair) while ind1 == ind2: ind1 = np.random.randint(10) ind2 = np.random.randint(10) # read the two images img1 = read_image('data/orl_faces/s' + str(i+1) + '/' + str(ind1 + 1) + '.pgm', 'rw+') img2 = read_image('data/orl_faces/s' + str(i+1) + '/' + str(ind2 + 1) + '.pgm', 'rw+') #reduce the size img1 = img1[::size, ::size] img2 = img2[::size, ::size] #store the images to the initialized numpy array x_geuine_pair[count, 0, 0, :, :] = img1 x_geuine_pair[count, 1, 0, :, :] = img2 #as we are drawing images from the same directory we assign label as 1. (genuine pair) y_genuine[count] = 1 count += 1 count = 0 x_imposite_pair = np.zeros([total_sample_size, 2, 1, dim1, dim2]) y_imposite = np.zeros([total_sample_size, 1]) for i in range(int(total_sample_size/10)): for j in range(10): #read images from different directory (imposite pair) while True: ind1 = np.random.randint(40) ind2 = np.random.randint(40) if ind1 != ind2: break img1 = read_image('data/orl_faces/s' + str(ind1+1) + '/' + str(j + 1) + '.pgm', 'rw+') img2 = read_image('data/orl_faces/s' + str(ind2+1) + '/' + str(j + 1) + '.pgm', 'rw+') img1 = img1[::size, ::size] img2 = img2[::size, ::size] x_imposite_pair[count, 0, 0, :, :] = img1 x_imposite_pair[count, 1, 0, :, :] = img2 #as we are drawing images from the different directory we assign label as 0. (imposite pair) y_imposite[count] = 0 count += 1 #now, concatenate, genuine pairs and imposite pair to get the whole data X = np.concatenate([x_geuine_pair, x_imposite_pair], axis=0)/255 Y = np.concatenate([y_genuine, y_imposite], axis=0) return X, Y Now, we generate our data and check our data size. As you can see, we have 20,000 data points and, out of these, 10,000 are genuine pairs and 10,000 are imposite pairs: X, Y = get_data(size, total_sample_size) X.shape (20000, 2, 1, 56, 46) Y.shape (20000, 1) Next, we split our data for training and testing with 75% training and 25% testing proportions: x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=.25) Now that we have successfully generated our data, we build our siamese network. First, we define the base network, which is basically a convolutional network used for feature extraction. We build two convolutional layers with ReLU activations and max pooling followed by a flat layer: def build_base_network(input_shape): seq = Sequential() nb_filter = [6, 12] kernel_size = 3 #convolutional layer 1 seq.add(Convolution2D(nb_filter[0], kernel_size, kernel_size, input_shape=input_shape, border_mode='valid', dim_ordering='th')) seq.add(Activation('relu')) seq.add(MaxPooling2D(pool_size=(2, 2))) seq.add(Dropout(.25)) #convolutional layer 2 seq.add(Convolution2D(nb_filter[1], kernel_size, kernel_size, border_mode='valid', dim_ordering='th')) seq.add(Activation('relu')) seq.add(MaxPooling2D(pool_size=(2, 2), dim_ordering='th')) seq.add(Dropout(.25)) #flatten seq.add(Flatten()) seq.add(Dense(128, activation='relu')) seq.add(Dropout(0.1)) seq.add(Dense(50, activation='relu')) return seq Next, we feed the image pair to the base network, which will return the embeddings, that is, feature vectors: input_dim = x_train.shape[2:] img_a = Input(shape=input_dim) img_b = Input(shape=input_dim) base_network = build_base_network(input_dim) feat_vecs_a = base_network(img_a) feat_vecs_b = base_network(img_b) feat_vecs_a and feat_vecs_b are the feature vectors of our image pair. Next, we feed these feature vectors to the energy function to compute the distance between them, and we use Euclidean distance as our energy function: def euclidean_distance(vects): x, y = vects return K.sqrt(K.sum(K.square(x - y), axis=1, keepdims=True)) def eucl_dist_output_shape(shapes): shape1, shape2 = shapes return (shape1[0], 1) distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([feat_vecs_a, feat_vecs_b]) Now, we set the epoch length to 13, and we use the RMS prop for optimization and define our model: epochs = 13 rms = RMSprop() model = Model(input=[input_a, input_b], output=distance) Next, we define our loss function as the contrastive_loss function and compile the model: def contrastive_loss(y_true, y_pred): margin = 1 return K.mean(y_true * K.square(y_pred) + (1 - y_true) * K.square(K.maximum(margin - y_pred, 0))) model.compile(loss=contrastive_loss, optimizer=rms) Now, we train our model: img_1 = x_train[:, 0] img_2 = x_train[:, 1] model.fit([img_1, img_2], y_train, validation_split=.25, batch_size=128, verbose=2, nb_epoch=epochs) You can see how the loss decreases over epochs: Train on 11250 samples, validate on 3750 samples Epoch 1/13 - 60s - loss: 0.2179 - val_loss: 0.2156 Epoch 2/13 - 53s - loss: 0.1520 - val_loss: 0.2102 Epoch 3/13 - 53s - loss: 0.1190 - val_loss: 0.1545 Epoch 4/13 - 55s - loss: 0.0959 - val_loss: 0.1705 Epoch 5/13 - 52s - loss: 0.0801 - val_loss: 0.1181 Epoch 6/13 - 52s - loss: 0.0684 - val_loss: 0.0821 Epoch 7/13 - 52s - loss: 0.0591 - val_loss: 0.0762 Epoch 8/13 - 52s - loss: 0.0526 - val_loss: 0.0655 Epoch 9/13 - 52s - loss: 0.0475 - val_loss: 0.0662 Epoch 10/13 - 52s - loss: 0.0444 - val_loss: 0.0469 Epoch 11/13 - 52s - loss: 0.0408 - val_loss: 0.0478 Epoch 12/13 - 52s - loss: 0.0381 - val_loss: 0.0498 Epoch 13/13 - 54s - loss: 0.0356 - val_loss: 0.0363 Now, we make predictions with test data: pred = model.predict([x_test[:, 0], x_test[:, 1]]) Next, we define a function for computing accuracy: def compute_accuracy(predictions, labels): return labels[predictions.ravel() < 0.5].mean() Now, we compute the accuracy of model: compute_accuracy(pred, y_test) 0.9779092702169625 In this tutorial, we have learned to build face recognition models using siamese networks. The architecture of siamese networks, basically consists of two identical neural networks both having the same weights and architecture and the output of these networks is plugged into some energy function to understand the similarity. To learn more about meta-learning with Python, check out the book Hands-On Meta-Learning with Python. What is Meta-Learning? Introducing Open AI’s Reptile: The latest scalable meta-learning Algorithm on the block “Deep meta reinforcement learning will be the future of AI where we will be so close to achieving artificial general intelligence (AGI)”, Sudharsan Ravichandiran

REST is an architectural style for implementing communication between the application client and server over HTTP. RESTful APIs use HTTP verbs (POST, GET, PUT, DELETE, and so on) to dictate the operation to be performed (Create, Read, Update, Delete) by the server on the domain entity. The REST style has become the de facto standard for creating services in modern application development. This makes it easy to use and consume services in any technology and on any platform, such as web frontends, desktop applications, or other web services. This article is an excerpt from a book written by Tamir Dresher, Amir Zuker, and Shay Friedman titled Hands-On Full-Stack Web Development with ASP.NET Core. In this book, you will learn how to build RESTful APIs in C# with ASP.NET Core, web APIs, and Entity Framework. Overview — REST APIs with ASP.NET Core API A basic ASP.NET Core MVC application can be broken down into three layers: models, controllers, and views. RESTful APIs in ASP.NET Core work very similarly; the only difference is that, instead of returning responses as visual views, the API response is a payload of data (usually in JSON format). The data returned from the API is later consumed by clients, such as Angular-based applications that can render the data as views, or by headless clients that have no UI and simply process data. For example, consider a background process that periodically sends notifications to a user about their account status: Before ASP.NET Core, Microsoft created an explicit distinction between ASP.NET MVC and the ASP.NET Web API. The former was used to create web applications with views that are generated by the server, while the former was used to create services that contain only logic and can be consumed by any client. Over time, the distinction between the two frameworks caused duplication of code and added a burden on the developers who needed to learn and master two technologies. ASP.NET Core unified the two frameworks into the ASP.NET Core MVC suite, and made it simpler to create web applications, with or without visual responses. Let's start with a simple API that will be the foundation for our application called, say, GiveNTake application. Creating a simple API The GiveNTake application allows the user to see a catalog of available products. For this to be possible, our server needs to include a specific API method that the client application can call and get back the collection of available products. We will treat the product as a simple string in the format of [Product ID] - [Product Name]: Open the any basic project you may have created, and add a new controller class to the Controllers folder.  Right-click on the newly created Controllers folder and choose Add | Controller. In the list of available templates, choose API Controller - Empty and click Add: Set the name to ProductsController and click Add. Add the following method to the generated controller: public string[] GetProducts() { return new[] { "1 - Microwave", "2 - Washing Machine", "3 - Mirror" }; } Your controller should look like this: [Route("api/Products")] [ApiController] public class ProductsController : Controller { public string[] GetProducts() { return new[] { "1 - Microwave", "2 - Washing Machine", "3 - Mirror" }; } } Congratulations! You have completed coding your first API method. The GetProducts method returns the collection of the available products. To see it in action, build and run your project. This will open a browser with the base address of your ASP.NET application. Add the  /api/Products string to the base address in the browser's address bar and execute it. You should see the collection of strings appear on the screen like so: Inspecting your APIs using Fiddler and Postman Using your browser to execute your APIs is nice enough for simple APIs that retrieve data, but as you go along and extend your APIs, you'll soon find that you need other powerful tools to test and debug what you develop. There are many tools that let you inspect and debug your APIs, but I have chosen to teach you about Fiddler and Postman because they are both simple and powerful. Fiddler Fiddler is a free web debugging tool that works as a proxy, logging all HTTP(S) traffic that is executed by processes in your computer. Fiddler allows you to inspect the traffic to see that exact HTTP request that was sent and the exact HTTP response that was returned. You can also use other advanced features, such as setting breakpoints and overriding the data that is sent or received. To install Fiddler, navigate to https://www.telerik.com/fiddler and click the Free download button. Save and run the installer: The Fiddler main screen is built from these main parts: Sessions list: Shows the HTTP(S) requests that were sent from processes in your machine Fiddler tabs: Contains different tools for inspecting and controlling sessions Request inspector: When the Inspectors tab and inner Raw tab are selected, this section shows the request as it was sent over-the-wire Response inspector: When the Inspectors tab and inner Raw tab are selected, this section shows the response as it was sent over-the-wire Immediately after you run Fiddler, it starts collecting the HTTP sessions that are performed in your machine. If you refresh the browser that you used to navigate to the /api/Products API  you created, you should see this session in Fiddler's Sessions List, as shown in the preceding screenshot. If you run a .NET application that sends HTTP requests to an address in your localhost, you won't see the session appear in Fiddler. Changing the address to localhost.fiddler will force the request to be captured by Fiddler. Fiddler is a great tool for debugging the requests and responses that are made in your application, but it means that you need to have a client that sends those requests. Many times when debugging and experimenting with APIs, you want to create HTTP requests manually and inspect them. You can accomplish this task with Fiddler's Composer tab, but I want to teach you about another tool that is much more suitable for these scenarios—Postman. Postman Postman is an HTTP client that simplifies the testing of web services and RESTful APIs. Postman allows you to easily construct HTTP requests, send them, and inspect them. Download Postman from https://www.getpostman.com/, and then install and run it: On the introduction screen, click on the Request option: Enter GetProducts in the Request name field, and then type GiveNTake into the collection section and create a new collection. Press Save to create the new request: Enter the full URL of the GetProducts API (for example, http://localhost:5267/api/products) in the URL field: Make sure that the HTTP Verb is set to GET and click on the Send button. After a few moments, you should see the response that was received from your service, and you can now inspect the status code, response body, and headers: You will find Postman to be an indispensable development tool while you develop your APIs, and we will use it many times as we go deeper into ASP.NET Core in this book. At this point, you might be wondering, how come the GetProducts method was invoked when we navigated to /api/Products? To answer this question, we need to talk about how ASP.NET Core routes requests into controllers and actions. ASP.NET Core provides the necessary infrastructure you need to create powerful RESTful APIs. In this article, you learned how to create controllers and actions that respond to HTTP requests, and return HTTP responses that you control. We've introduced two popular tools: Fiddler and Postman, and you'll find them very useful when you create and debug your API applications. To know more about APIs in ASP.NET Core, check out the book Hands-On Full-Stack Web Development with ASP.NET Core. Google announces the general availability of a new API for Google Docs What to expect in ASP.NET Core 3.0 How to call an Azure function from an ASP.NET Core MVC application

JavaScript, being a flexible scripting language along with its dynamic type system, can become harder to maintain, the more a project scales up and as the team's staff changes. There are many tools and languages that can assist with this situation, one of which is TypeScript. This article is an excerpt from a book written by Tamir Dresher, Amir Zuker, and Shay Friedman titled Hands-On Full-Stack Web Development with ASP.NET Core. In this book, you will learn how to build web applications using Angular, React, and Vue. Back in the day when JavaScript ES5 was the active version, writing JavaScript code that adhered to encapsulation and modularity was not a trivial task. Things such as prototypes, namespaces, and self-executing functions could certainly improve your code; unfortunately, many JavaScript developers did not invest the necessary effort to improve the manageability of their code using those constructs. Consequently, new technologies surfaced to help with this matter; a popular example relevant at that time was CoffeeScript. Then, JavaScript ES6 was released. ES6, with its great added features, such as modules, classes, and more specific variable scoping, basically overturned CoffeeScript and similar alternatives at that time. Evidently, JavaScript is still missing a key feature, and that is type information. JavaScript's dynamic type system is a strong feature at times; unfortunately, the manageability and reusability of existing code suffers due to this fact. For that key purpose, every large-scale project should examine the use of tools that compensate the situation; those leading the field now are TypeScript and Flow. TypeScript, an open source scripting language, is developed and maintained by Microsoft. Its killer feature is bringing static typing to JavaScript-based systems and allowing you to annotate the code with type information. Additionally, it supports augmenting existing JavaScript code with external type information, similar to the header files pattern, as it's commonly known in C++. TypeScript is a superset of JavaScript, that meaning it has seamless integration with JavaScript, and makes JavaScript developers feel right at home. Being a JavaScript developer makes you a TypeScript developer as well since JavaScript code is actually valid TypeScript. Importantly, this makes the gradual upgrade of existing JavaScript code fairly easy. Another example of how great TypeScript is is the fact that Angular, a framework built by Google, adopts it extensively, and is actually implemented in TypeScript internally. The TypeScript compiler TypeScript is a superset of JavaScript, yet browsers and other platforms don't recognize it, nor are they able to execute it. For that purpose, TypeScript is processed by the TypeScript compiler, which transpiles your TypeScript code to JavaScript. Then, you can run the transpiled code basically everywhere JavaScript is supported. The tsc is available as an npm package that you can install globally. Open your favorite terminal and install it using the following command: npm install -g typescript Afterward, you can use the command tsc to execute the tsc: TypeScript files have a .ts file extension; this is where you write your TypeScript code. When needed, you use the TypeScript compiler to transpile your TypeScript code, which creates the JavaScript implementation of the code; you do that by executing the following command: tsc <typescript_filename> Types in Typescript One of the key features of TypeScript is bringing static typing to JavaScript. In TypeScript, you annotate your code with type information; every declaration can be defined with its associated typing. In the declaration, you do that by adding a colon following the type. For example, the following statement defines a variable of type number: const productsCount: number; Annotating your code with type information enables validation. The TypeScript compiler should throw errors if you try to do anything that is not supported by the specified type, thus productivity and discoverability benefit substantially through type safety. Additionally, with proper tooling support, you get IntelliSense support and automatic code completion. TypeScript supports a handful of types to reflect most constructs in JavaScript. We will cover these next, starting with basic types. Basic types TypeScript supports a handful of built-in types; the following are examples of the most common ones: const fullName: string = "John Doe"; const age: number = 6; const isDone: boolean = false; const d: Date = new Date(); const canBeAnything: any = 6; As you can see, TypeScript supports basic primitive types such as string, number, boolean, and date. Another thing to notice is the type any. The any type is a valid TypeScript type that indicates that the declaration can be of any given type and all operations on it should be allowed and considered safe. Arrays TypeScript arrays can be written in one of two ways. You can specify the item type followed by square brackets or you can use the generic Array type as follows: const list: number[] = [1, 2, 3]; const list: Array<number> = [1, 2, 3]; Enums Enums allow you to specify named values that correspond to numeric or string values. If you don't manually set the associated corresponding value, the named values in the enum are numbered, starting at 0 by default. Enums are extremely useful, and are commonly used to represent a set of predefined options or lookup values. In TypeScript, you define enums by using the enum keyword as follows: enum Color {Red, Green, Blue} const c: Color = Color.Red; Objects In JavaScript, objects contain key/value pairs that form the shape of the object. TypeScript supports object types as well, very similar to how you write plain JavaScript objects. Consider the following plain JavaScript object: const obj = { x: 5, y: 6 } In the preceding code, obj is a plain object defined with two keys, x and y, both with number values. To define an object type in TypeScript, you use a similar format — curly braces to represent an object, inside the keys, and their type information: { x:number, y:number } Together, this is how you associate a type to an object: const obj: { x:number, y:number } = { x: 5, y: 6 } In the preceding code, the obj object is initialized the same as before, only now along with its type information, which is marked in bold. Functions Functions have type information as well. Functions are comprised of parameters and a return value: function buildName(firstName: string, lastName: string): string { return firstName + " " + lastName; } Each parameter is annotated with its type, and then the return value's type is added at the end of the function signature. Unlike JavaScript, TypeScript enforces calls to functions to adhere to the defined signature. For example, if you try to call the function with anything other than two string parameters, the TypeScript compiler will not allow it. You can define more flexible function signatures if you like. TypeScript supports defining optional parameters by using the question mark symbol, ?: function buildName(firstName: string, lastName: string, title?: string): string { return title + " " + firstName + " " + lastName; } const name = buildName('John', 'Doe'); // valid call The preceding function can now be called with either 2 or 3 string parameters. Everything in JavaScript is still supported, of course. You can still use ES6 default parameter values and rest parameters as well. Type inference Until now, the illustrated code included the type information in the same statement where the actual value assignment took place. TypeScript supports type inference, meaning that it tries to resolve the relevant type information if you don't specify any. Let's look at some examples: const count = 2; In the preceding simple example, TypeScript is smart enough to realize that the type of count is a number without you having to explicitly specify that: function sum(x1: number, x2: number) { return x1 + x2; } const mySum = sum(1, 2); The preceding example demonstrates the power of type inference in TypeScript. If you pay close attention, the code doesn't specify the return value type of the sum function. TypeScript evaluates the code and determines that the return type is number in this case. As a result, the type of the variable mySum is also a number. Type inference is an extremely useful feature since it saves us the incredible time that would be otherwise spent in adding endless type information. Type casting Having the code statically typed leads to type safety. TypeScript does not let you perform actions that are not considered safe and supported. In the following example, the loadProducts function is defined with a return value of type any, thus the type of products is inferred as any: function loadProducts(): any { return [{name: 'Book1'}, {name: 'Book2'}]; } const products = loadProducts(); // products is of type 'any' In some cases, you may actually know the expected type. You can instruct TypeScript of the type by using a cast. Typecasting in TypeScript is supported by using the as an operator or the use of angle brackets, as follows: const products = loadProducts() as {name: string}[]; const products = <{name: string}[]>loadProducts(); Using the preceding cast, you have full support and recognition by TypeScript that products is now an array of objects with a name key. Type aliasing TypeScript supports defining new types via type aliases. You can use type aliases to reuse type information in multiple declarations, for example: type person = {name: string}; let employee: person; let contact: person; Given the preceding example, the code defines a type alias named person as an object with a name key of type string. Afterwards, the rest of the code can reference this type where needed. Type aliases are somewhat similar to interfaces (covered next), but can name primitives, unions, and tuples. Unlike interfaces, type aliases cannot be extended or implemented from, and so interfaces are generally preferred. Unions and intersections allow you to construct a type from multiple existing types, while tuples express arrays with a fixed number of known elements. You can read more about these at https://www.typescriptlang.org/docs/handbook/basic-types.html and https://www.typescriptlang.org/docs/handbook/advanced-types.html. TypeScript is one of the most popular scripting languages in respect to JavaScript-based code bases. At its very core, it brings static typing and assists tremendously with maintainability and productivity. In this article, we covered TypeScript types. TypeScript advances at a steady pace and shows great promise for using it in large projects. It has more features that weren't addressed in this tutorial, such as generics, mapped types, abstract classes, project references, and much more that will help you for ASP.NET core development. To know more about interfaces and compilers of TypeScript, check out the book Hands-On Full-Stack Web Development with ASP.NET Core. Typescript 3.3 is finally released! Future of ESLint support in TypeScript Introducing ReX.js v1.0.0 a companion library for RegEx written in TypeScript

In a paper published on February 4, Google engineers drafted out plans to forward federated learning at a scale. It showcases the high-level plans, challenges, solutions, and applications. Federated learning was first introduced in 2017 by Google. The idea is to use data from a number of computing devices like smartphones instead of a centralized data source. Federated learning can help with privacy Federated learning can be beneficial as it addresses the privacy concern. Android phones are used for the system where the data is only used but never uploaded to any server. A deep neural network is trained by using TensorFlow on the data stored in the Android phone. The Federated averaging algorithm by Brendan McMahan uses a similar approach as synchronous training. The weights of the neural network are combined in the cloud using Federated Averaging. This creates a global model which is then pushed back to the phones as results/desirable actions. To enhance privacy approaches like differential privacy and Secure aggregation are taken. The paper addresses challenges like time zone differences, connectivity issues, interrupted execution etc,. Their work is mature enough to deploy the system in production for tens of millions of devices. They are working towards supporting billions of devices now. The training protocol The system involves devices and the Federated Learning server communicating availability and the server selecting devices to run a task. A subset of the available devices are selected for a task. The Federated Learning server instructs the devices what computing task to run with a plan. A plan would consist a TensorFlow graph and instructions to execute it. There are three phases for the training to take place: Selection of the devices that meet eligibility criteria Configuring the server with simple or Secure Aggregation Reporting from the devices where reaching a certain number would get the training round started Source: Towards Federated Learning at Scale: System Design The devices are supposed to maintain a repository of the collected data and the applications are responsible to provide data to the Federated Learning runtime as an example store. The Federated Learning server is designed to operate on orders of many magnitudes. Each round can mean updates from devices in the range of KBs to tens of MBs coming going the server. Data collection To avoid harming the phone’s battery life and performance, various analytics are collected in the cloud. The logs don’t contain any personally identifiable information. Secure aggregation Secure aggregation uses encryption to make individual device updates uninspectable. They plant to use it for protection against threats in data centers. Secure aggregation would ensure data encryption even when it is in-memory. Challenges of federated learning Compared to a centralized dataset, federated learning poses a number of challenges. The training data is not inspectable, tooling is required to work with proxy data. Models cannot be run interactively and must be compiled to be deployed in the Federated Learning server. Model resource consumption and runtime compatibility also come into the picture when working with many devices in real-time. Applications of Federated Learning It is best for cases where the data on devices is more relevant than data on servers. Ranking items for better navigation, suggestions for on-device keyboard, and next word prediction. This has already been implemented on Google pixel and Gboard. Future work is to eliminate bias caused be restrictions in device selection, algorithms to support better parallelism (more devices in one round), avoiding retraining already trained tasks on devices, and compression to save bandwidth. Federated computation, not federated learning The authors do no mention machine learning explicitly anywhere in the paper. They believe that the applications of such a model are not limited to machine learning. Federated Computation is the term they want to use for this concept. Federated computation and edge computing Federated learning and edge computing are very similar, there are but subtle differences in the purpose of these two. Federated learning is used to solve problems with specific tasks assigned to endpoint smartphones. Edge computing is for predefined tasks to be processed at end nodes, for example, IoT cameras. Federated learning decentralizes the data used while edge computing decentralizes the task computation to various devices. For more details on the architecture and its working, you can check out the research paper. Technical and hidden debts in machine learning – Google engineers’ give their perspective Researchers introduce a machine learning model where the learning cannot be proved What if AIs could collaborate using human-like values? DeepMind researchers propose a Hanabi platform.

Open source software in general, and JavaScript in particular, can seem like a place where boom and bust is the rule of law: rapid growth before everyone moves on to the next big thing. But Node.js is different. Although it certainly couldn’t be described as new, and it's growth hasn't been dramatic by any measure, over the last few years it has managed to push itself forward as one of the most widely used JavaScript tools on the planet. Do you want to learn Node.js? Popularity, however can only tell you so much. The key question, if you’re reading this, is whether you should learn Node.js. So, to help you decide if it’s time to learn the JavaScript library, here’s a list of the biggest reasons why you should start learning Node.js... Learn everything you need to know about Node.js with Packt's Node.js Complete Reference Guide Book Learning Path. Node.js lets you write JavaScript on both client and server Okay, let’s get the obvious one out of the way first: Node.js is worth learning because it allows you to write JavaScript on the server. This has arguably transformed the way we think about JavaScript. Whereas in the past it was a language specifically written on the client, backed by the likes of PHP and Java, it’s now a language that you can use across your application. Read next: The top 5 reasons Node.js could topple Java This is important because it means teams can work much more efficiently together. Using different languages for backend and frontend is typically a major source of friction. Unless you have very good polyglot developers, a team is restricted to their core skills, while tooling is also more inflexible. If you’re using JavaScript across the stack, it’s easier to use a consistent toolchain. From a personal perspective, learning Node.js is a great starting point for full stack development. In essence, it's like an add-on that immediately expands what you can do with JavaScript. In terms of your career, then, it could well make you an invaluable asset to a development team. Read next: How is Node.js changing web development? Node.js allows you to build complex and powerful applications without writing complex code Another strong argument for Node.js is that it is built for performance. This is because of 2 important things - Node.js' asynchronous-driven architecture, and the fact that it uses the V8 JavaScript engine. The significance of this is that V8 is one of the fastest implementations of JavaScript, used to power many of Google’s immensely popular in-browser products (like Gmail). Node.js is powerful because it employs an asynchronous paradigm for handling data between client and server. To clarify what this means, it’s worth comparing to the typical application server model that uses blocking I/O - in this instance, the application has to handle each request sequentially, suspending threads until they can be processed. This can add complexity to an application and, of course, slows an application down. In contrast, Node.js allows you to use non-blocking I/O in which threads (in this case sequential, not concurrent), which can manage multiple requests. If one can’t be processed, it’s effectively ‘withheld’ as a promise, which means it can be executed later without holding up other threads. This means Node.js can help you build applications of considerable complexity without adding to the complexity of your code. Node.js is well suited to building microservices Microservices have become a rapidly growing architectural style that offer increased agility and flexibility over the traditional monolith. The advantages of microservices are well documented, and whether or not they’re right for you now, it’s likely that they’re going to dominate the software landscape as the world moves away from monolithic architecture. This fact only serves to strengthen the argument that you should learn Node.js because the library is so well suited to developing in this manner. This is because it encourages you to develop in a modular and focused manner, quite literally using specific modules to develop an application. This is distinct and almost at odds with the monolithic approach to software architecture. At this point, it’s probably worth highlighting that it’s incredibly easy to package and publish the modules you build thanks to npm (node package manager). So, even if you haven’t yet worked with microservices, learning Node.js is a good way to prepare yourself for a future where they are going to become even more prevalent. Node.js can be used for more than just web development We know by now that Node.js is flexible. But it’s important to recognise that its flexibility means it can be used for a wide range of different purposes. Yes, the library's community are predominantly building applications for the web, but it’s also a useful tool for those working in ops or infrastructure. This is because Node.js is a great tool for developing other development tools. If you’re someone working to support a team of developers, or, indeed, to help manage an entire distributed software infrastructure, it could be vital in empowering you to get creative and build your own support tools. Even more surprisingly, Node.js can be used in some IoT projects. As this post from 2016 suggests, the two things might not be quite such strange bedfellows. Node.js is a robust project that won't be going anywhere As I’ve already said, in the JavaScript world frameworks and tools can appear and disappear quickly. That means deciding what to learn, and, indeed, what to integrate into your stack, can feel like a bit of a gamble. However, you can be sure that Node.js is here to stay. There are a number of reasons for this. For starters, there’s no other tool that brings JavaScript to the server. But more than that, with Google betting heavily on V8 - which is, as we’ve seen, such an important part of the project - you can be sure it’s only going to go from strength to strength. It’s also worth pointing out that Node.js went through a small crisis when io.js broke away from the main Node.js project. This feud was as much personal as it was technical, but with the rift healed, and the Node.js Foundation now managing the whole project, helping to ensure that the software is continually evolving with other relevant technological changes and that the needs of the developers who use it continue to be met. Conclusion: spend some time exploring Node.js before you begin using it at work That’s just 5 reasons why you should learn Node.js. You could find more, but broadly speaking these all underline its importance in today’s development world. If you’re still not convinced, there’s a caveat. If Node.js isn’t yet right for you, don’t assume that it’s going to fix any technological or cultural issues that have been causing you headaches. It probably won’t. In fact, you should probably tackle those challenges before deciding to use it. But that all being said, even if you don’t think it’s the right time to use Node.js professionally, that doesn’t mean it isn’t worth learning. As you can see, it’s well worth your time. Who knows where it might take you? Ready to begin learning? Purchase Node.js Complete Reference Guide or read it for free with a subscription free trial.

In this article, you will learn how to use lambda expressions in Java 11. This article is an excerpt from a book written by Nick Samoylov and Mohamed Sanaulla titled Java 11 Cookbook - Second Edition. In this book, you will learn how to build graphical user interfaces using JavaFX. Getting ready Creating and using lambda expressions is actually much simpler than writing a method. One just needs to list the input parameters, if any, and the code that does what has to be done. Let's see an implementation of standard functional interfaces rewritten using lambda expressions. Here's how we have implemented the four main functional interfaces using anonymous classes: Function<Integer, Double> ourFunc = new Function<Integer, Double>(){ public Double apply(Integer i){ return i * 10.0; } }; System.out.println(ourFunc.apply(1)); //prints: 10.0 Consumer<String> consumer = new Consumer<String>() { public void accept(String s) { System.out.println("The " + s + " is consumed."); } }; consumer.accept("Hello!"); //prints: The Hello! is consumed. Supplier<String> supplier = new Supplier<String>() { public String get() { String res = "Success"; //Do something and return result—Success or Error. return res; } }; System.out.println(supplier.get()); //prints: Success Predicate<Double> pred = new Predicate<Double>() { public boolean test(Double num) { System.out.println("Test if " + num + " is smaller than 20"); return num < 20; } }; System.out.println(pred.test(10.0)? "10 is smaller":"10 is bigger"); //prints: Test if 10.0 is smaller than 20 // 10 is smaller And here's how they look with lambda expressions: Function<Integer, Double> ourFunc = i -> i * 10.0; System.out.println(ourFunc.apply(1)); //prints: 10.0 Consumer<String> consumer = s -> System.out.println("The " + s + " is consumed."); consumer.accept("Hello!"); //prints: The Hello! is consumed. Supplier<String> supplier = () - > { String res = "Success"; //Do something and return result—Success or Error. return res; }; System.out.println(supplier.get()); //prints: Success Predicate<Double> pred = num -> { System.out.println("Test if " + num + " is smaller than 20"); return num < 20; }; System.out.println(pred.test(10.0)? "10 is smaller":"10 is bigger"); //prints: Test if 10.0 is smaller than 20 // 10 is smaller The examples of specialized functional interfaces we have presented are as follows: IntFunction<String> ifunc = new IntFunction<String>() { public String apply(int i) { return String.valueOf(i * 10); } }; System.out.println(ifunc.apply(1)); //prints: 10 BiFunction<String, Integer, Double> bifunc = new BiFunction<String, Integer, Double >() { public Double apply(String s, Integer i) { return (s.length() * 10d) / i; } }; System.out.println(bifunc.apply("abc",2)); //prints: 15.0 BinaryOperator<Integer> binfunc = new BinaryOperator<Integer>(){ public Integer apply(Integer i, Integer j) { return i >= j ? i : j; } }; System.out.println(binfunc.apply(1,2)); //prints: 2 IntBinaryOperator intBiFunc = new IntBinaryOperator(){ public int applyAsInt(int i, int j) { return i >= j ? i : j; } }; System.out.println(intBiFunc.applyAsInt(1,2)); //prints: 2 And here's how they look with lambda expressions: IntFunction<String> ifunc = i -> String.valueOf(i * 10); System.out.println(ifunc.apply(1)); //prints: 10 BiFunction<String, Integer, Double> bifunc = (s,i) -> (s.length() * 10d) / i; System.out.println(bifunc.apply("abc",2)); //prints: 15.0 BinaryOperator<Integer> binfunc = (i,j) -> i >= j ? i : j; System.out.println(binfunc.apply(1,2)); //prints: 2 IntBinaryOperator intBiFunc = (i,j) -> i >= j ? i : j; System.out.println(intBiFunc.applyAsInt(1,2)); //prints: 2 As you can see, the code is less cluttered and more readable. How to form lambda expressions Those who have some traditional code-writing experience, when starting functional programming, equate functions with methods.  They try to create functions first because that was how we all used to write traditional code—by creating methods. Yet, functions are just smaller pieces of functionality that modify some aspects of the behavior of the methods or provide the business logic for the otherwise non-business-specific code. In functional programming, as in traditional programming, methods continue to provide the code structure, while functions are the nice and helpful additions to it. So, in functional programming, creating a method comes first, before the functions are defined. Let's demonstrate this. The following are the basic steps of code writing. First, we identify the well-focused block of code that can be implemented as a method. Then, after we know what the new method is going to do, we can convert some pieces of its functionality into functions: Create the calculate() method: void calculate(){ int i = 42; //get a number from some source double res = 42.0; //process the above number if(res < 42){ //check the result using some criteria //do something } else { //do something else } } The preceding pseudocode outlines the idea of the calculate() method's functionality. It can be implemented in a traditional style—by using methods, as follows: int getInput(){ int result; //getting value for result variable here return result; } double process(int i){ double result; //process input i and assign value to result variable } boolean checkResult(double res){ boolean result = false; //use some criteria to validate res value //and assign value to result return result; } void processSuccess(double res){ //do something with res value } void processFailure(double res){ //do something else with res value } void calculate(){ int i = getInput(); double res = process(i); if(checkResult(res)){ processSuccess(res); } else { processFailure(res); } } But some of these methods may be very small, so the code becomes fragmented and less readable with so many additional indirections. This disadvantage becomes especially glaring in the case when the methods come from outside the class where the calculate() method is implemented: void calculate(){ SomeClass1 sc1 = new SomeClass1(); int i = sc1.getInput(); SomeClass2 sc2 = new SomeClass2(); double res = sc2.process(i); SomeClass3 sc3 = new SomeClass3(); SomeClass4 sc4 = new SomeClass4(); if(sc3.checkResult(res)){ sc4.processSuccess(res); } else { sc4.processFailure(res); } } As you can see, in the case where each of the external methods is small, the amount of plumbing code may substantially exceed the payload it supports. Besides, the preceding implementation creates many tight dependencies between classes. Let's look at how we can implement the same functionality using functions. The advantage is that the functions can be as small as they need to be, but the plumbing code will never exceed the payload because there is no plumbing code. Another reason to use functions is when we need the flexibility to change sections of the functionality on the fly, for the algorithm's research purpose. And if these pieces of functionality have to come from outside the class, we do not need to build other classes just for the sake of passing a method into calculate(). We can pass them as functions: void calculate(Supplier<Integer> souc e, Function<Integer, Double> process, Predicate<Double> condition, Consumer<Double> success, Consumer<Double> failure){ int i = source.get(); double res = process.apply(i); if(condition.test(res)){ success.accept(res); } else { failure.accept(res); } } Here's how the functions may look: Supplier<Integer> source = () -> 4; Function<Integer, Double> before = i -> i * 10.0; Function<Double, Double> after = d -> d + 10.0; Function<Integer, Double> process = before.andThen(after); Predicate<Double> condition = num -> num < 100; Consumer<Double> success = d -> System.out.println("Success: "+ d); Consumer<Double> failure = d -> System.out.println("Failure: "+ d); calculate(source, process, condition, success, failure); The result of the preceding code is going to be as follows: Success: 50.0 How lambda expressions work The lambda expression acts as a regular method, except when you think about testing each function separately. How to do it? There are two ways to address this issue. First, since the functions are typically small, there is often no need to test them separately, and they are tested indirectly when the code that uses them is tested. Second, if you still think the function has to be tested, it is always possible to wrap it in the method that returns the function, so you can test that method like any other method. Here is an example of how it can be done: public class Demo { Supplier<Integer> source(){ return () -> 4;} Function<Double, Double> after(){ return d -> d + 10.0; } Function<Integer, Double> before(){return i -> i * 10.0; } Function<Integer, Double> process(){return before().andThen(after());} Predicate<Double> condition(){ return num -> num < 100.; } Consumer<Double> success(){ return d -> System.out.println("Failure: " + d); } Consumer<Double> failure(){ return d-> System.out.println("Failure: " + d); } void calculate(Supplier<Integer> souce, Function<Integer, Double> process, Predicate<Double> condition, Consumer<Double> success, Consumer<Double> failure){ int i = source.get(); double res = process.apply(i); if(condition.test(res)){ success.accept(res); } else { failure.accept(res); } } void someOtherMethod() { calculate(source(), process(), condition(), success(), failure()); } Now we can write the function unit tests as follows: public class DemoTest { @Test public void source() { int i = new Demo().source().get(); assertEquals(4, i); } @Test public void after() { double d = new Demo().after().apply(1.); assertEquals(11., d, 0.01); } @Test public void before() { double d = new Demo().before().apply(10); assertEquals(100., d, 0.01); } @Test public void process() { double d = new Demo().process().apply(1); assertEquals(20., d, 0.01); } @Test public void condition() { boolean b = new Demo().condition().test(10.); assertTrue(b); } } Typically, lambda expressions (and functions in general) are used for specializing otherwise generic functionalities—by adding business logic to a method. A good example is stream operations. The library authors have created them to be able to work in parallel, which required a lot of expertise. And now the library users can specialize the operations by passing into them the lambda expressions (functions) that provide the application's business logic. Function inlining Since, as we have mentioned already, functions are often simple one-liners, they are often inlined when passed in as parameters, for example: Consumer<Double> success = d -> System.out.println("Success: " + d); Consumer<Double> failure = d-> System.out.println("Failure: " + d); calculate(() -> 4, i -> i * 10.0 + 10, n -> n < 100, success, failure); But one should not push it too far, as such inlining may decrease code readability. In this tutorial, we learned how to use lambda expressions in Java 11. To learn more Java 11 recipes, check out the book Java 11 Cookbook - Second Edition. Brian Goetz on Java futures at FOSDEM 2019 7 things Java programmers need to watch for in 2019 Clojure 1.10 released with Prepl, improved error reporting and Java compatibility

In this tutorial, we will learn new ways to interact with network devices using Python. We will understand how to configure network devices using configuration templates, and also write a modular code to ensure high reusability of the code to perform repetitive tasks. We will also see the benefits of parallel processing of tasks and the efficiency that can be gained through multithreading. This Python tutorial has been taken from the second edition of Practical Network Automation. Read it here. Interacting with network devices Python is widely used to perform network automation. With its wide set of libraries (such as Netmiko and Paramiko), there are endless possibilities for network device interactions for different vendors. Let us understand one of the most widely used libraries for network interactions. We will be using Netmiko to perform our network interactions. Python provides a well-documented reference for each of the modules, and, for our module, the documentation can be found at pypi.org.  For installation, all we have to do is go into the folder from the command line where python.exe is installed or is present. There is a subfolder in that location called scripts. Inside the folder, we have two options that can be used for installing the easy_install.exe or pip.exe modules. Installing the library for Python can be done in two ways: The syntax of easy_install is as follows: easy_install <name of module> For example, to install Netmiko, the following command is run: easy_install netmiko The syntax of pip install is as follows: pip install <name of module> For example: pip install netmiko Here's an example of a simple script to log in to the router (an example IP is 192.168.255.249 with a username and password of cisco) and show the version: from netmiko import ConnectHandler device = ConnectHandler(device_type='cisco_ios', ip='192.168.255.249', username='cisco', password='cisco') output = device.send_command("show version") print (output) device.disconnect() The output of the execution of code against a router is as follows: As we can see in the sample code, we call the ConnectHandler function from the Netmiko library, which takes four inputs (platform type, IP address of device, username, and password): Netmiko works with a variety of vendors. Some of the supported platform types and their abbreviations to be called in Netmiko are as follows: a10: A10SSH, accedian: AccedianSSH, alcatel_aos: AlcatelAosSSH, alcatel_sros: AlcatelSrosSSH, arista_eos: AristaSSH, aruba_os: ArubaSSH, avaya_ers: AvayaErsSSH, avaya_vsp: AvayaVspSSH, brocade_fastiron: BrocadeFastironSSH, brocade_netiron: BrocadeNetironSSH, brocade_nos: BrocadeNosSSH, brocade_vdx: BrocadeNosSSH, brocade_vyos: VyOSSSH, checkpoint_gaia: CheckPointGaiaSSH, ciena_saos: CienaSaosSSH, cisco_asa: CiscoAsaSSH, cisco_ios: CiscoIosBase, cisco_nxos: CiscoNxosSSH, cisco_s300: CiscoS300SSH, cisco_tp: CiscoTpTcCeSSH, cisco_wlc: CiscoWlcSSH, cisco_xe: CiscoIosBase, cisco_xr: CiscoXrSSH, dell_force10: DellForce10SSH, dell_powerconnect: DellPowerConnectSSH, eltex: EltexSSH, enterasys: EnterasysSSH, extreme: ExtremeSSH, extreme_wing: ExtremeWingSSH, f5_ltm: F5LtmSSH, fortinet: FortinetSSH, generic_termserver: TerminalServerSSH, hp_comware: HPComwareSSH, hp_procurve: HPProcurveSSH, huawei: HuaweiSSH, juniper: JuniperSSH, juniper_junos: JuniperSSH, linux: LinuxSSH, mellanox_ssh: MellanoxSSH, mrv_optiswitch: MrvOptiswitchSSH, ovs_linux: OvsLinuxSSH, paloalto_panos: PaloAltoPanosSSH, pluribus: PluribusSSH, quanta_mesh: QuantaMeshSSH, ubiquiti_edge: UbiquitiEdgeSSH, vyatta_vyos: VyOSSSH, vyos: VyOSSSH Depending upon the selection of the platform type, Netmiko can understand the returned prompt and the correct way to SSH into the specific device. Once the connection is made, we can send commands to the device using the send_command method. Once we get the return value, the value stored in the output variable is displayed, which is the string output of the command that we sent to the device. The last line, which uses the disconnect function, ensures that the connection is terminated cleanly once we are done with our task. For configuration (for example, we need to provide a description to the FastEthernet 0/0 router interface), we use Netmiko, as shown in the following example: from netmiko import ConnectHandler print ("Before config push") device = ConnectHandler(device_type='cisco_ios', ip='192.168.255.249', username='cisco', password='cisco') output = device.send_command("show running-config interface fastEthernet 0/0") print (output) configcmds=["interface fastEthernet 0/0", "description my test"] device.send_config_set(configcmds) print ("After config push") output = device.send_command("show running-config interface fastEthernet 0/0") print (output) device.disconnect() The output of the execution of the preceding code is as follows: As we can see, for config push, we do not have to perform any additional configurations but just specify the commands in the same order as we send them manually to the router in a list, and pass that list as an argument to the send_config_set function. The output in Before config push is a simple output of the FastEthernet0/0 interface, but the output under After config push now has the description that we configured using the list of commands. In a similar way, we can pass multiple commands to the router, and Netmiko will go into configuration mode, write those commands to the router, and exit config mode. If we want to save the configuration, we use the following command after the send_config_set command: device.send_command("write memory") This ensures that the router writes the newly pushed configuration in memory. Additionally, for reference purposes across the book, we will be referring to the following GNS3 simulated network: In this topology, we have connected four routers with an Ethernet switch. The switch is connected to the local loopback interface of the computer, which provides the SSH connectivity to all the routers. We can simulate any type of network device and create topology based upon our specific requirements in GNS3 for testing and simulation. This also helps in creating complex simulations of any network for testing, troubleshooting, and configuration validations. The IP address schema used is the following: rtr1: 192.168.20.1 rtr2: 192.168.20.2 rtr3: 192.168.20.3 rtr4: 192.168.20.4 Loopback IP of computer: 192.168.20.5 The credentials used for accessing these devices are the following: Username: test Password: test Let us start from the first step by pinging all the routers to confirm their reachability from the computer. The code is as follows: import socket import os s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) for n in range(1, 5): server_ip="192.168.20.{0}".format(n) rep = os.system('ping ' + server_ip) if rep == 0: print ("server is up" ,server_ip) else: print ("server is down" ,server_ip) The output of running the preceding code is as follows: As we can see in the preceding code, we use the range command to iterate over the IPs 192.168.20.1-192.168.20.4. The server_ip variable in the loop is provided as an input to the ping command, which is executed for the response. The response stored in the rep variable is validated with a value of 0 stating that the router can be reached, and a value of 1 means the router is not reachable. As a next step, to validate whether the routers can successfully respond to SSH, let us fetch the value of uptime from the show version command: show version | in uptime The code is as follows: from netmiko import ConnectHandler username = 'test' password="test" for n in range(1, 5): ip="192.168.20.{0}".format(n) device = ConnectHandler(device_type='cisco_ios', ip=ip, username='test', password='test') output = device.send_command("show version | in uptime") print (output) device.disconnect() The output of running the preceding command is as follows: Using Netmiko, we fetched the output of the command from each of the routers and printed a return value. A return value for all the devices confirms SSH attainability, whereas failure would have returned an exception, causing the code to abruptly end for that particular router. If we want to save the configuration, we use the following command after the send_config_set command: device.send_command("write memory") This ensures that the router writes the newly pushed configuration in memory. Network device configuration using template With all the routers reachable and accessible through SSH, let us configure a base template that sends the Syslog to a Syslog server and additionally ensures that only information logs are sent to the Syslog server. Also, after configuration, a validation needs to be performed to ensure that logs are being sent to the Syslog server. The logging server info is as follows: Logging server IP: 192.168.20.5  Logging port: 514 Logging protocol: TCP Additionally, a loopback interface (loopback 30) needs to be configured with the {rtr} loopback interface description. The code lines for the template are as follows: logging host 192.168.20.5 transport tcp port 514 logging trap 6 interface loopback 30 description "{rtr} loopback interface" To validate that the Syslog server is reachable and that the logs sent are informational, use the show logging command. In the event that  the output of the command contains the text: Trap logging: level informational: This confirms that the logs are sent as informational Encryption disabled, link up: This confirms that the Syslog server is reachable The code to create the configuration, push it on to the router and perform the validation, is as follows: from netmiko import ConnectHandler template="""logging host 192.168.20.5 transport tcp port 514 logging trap 6 interface loopback 30 description "{rtr} loopback interface\"""" username = 'test' password="test" #step 1 #fetch the hostname of the router for the template for n in range(1, 5): ip="192.168.20.{0}".format(n) device = ConnectHandler(device_type='cisco_ios', ip=ip, username='test', password='test') output = device.send_command("show run | in hostname") output=output.split(" ") hostname=output[1] generatedconfig=template.replace("{rtr}",hostname) #step 2 #push the generated config on router #create a list for generateconfig generatedconfig=generatedconfig.split("\n") device.send_config_set(generatedconfig) #step 3: #perform validations print ("********") print ("Performing validation for :",hostname+"\n") output=device.send_command("show logging") if ("encryption disabled, link up"): print ("Syslog is configured and reachable") else: print ("Syslog is NOT configured and NOT reachable") if ("Trap logging: level informational" in output): print ("Logging set for informational logs") else: print ("Logging not set for informational logs") print ("\nLoopback interface status:") output=device.send_command("show interfaces description | in loopback interface") print (output) print ("************\n") The output of running the preceding command is as follows: Another key aspect to creating network templates is understanding the type of infrastructure device for which the template needs to be applied. As we generate the configuration form templates, there are times when we want to save the generated configurations to file, instead of directly pushing on devices. This is needed when we want to validate the configurations or even keep a historic repository for the configurations that are to be applied on the router. Let us look at the same example, only this time, the configuration will be saved in files instead of writing back directly to routers. The code to generate the configuration and save it as a file is as follows: from netmiko import ConnectHandler import os template="""logging host 192.168.20.5 transport tcp port 514 logging trap 6 interface loopback 30 description "{rtr} loopback interface\"""" username = 'test' password="test" #step 1 #fetch the hostname of the router for the template for n in range(1, 5): ip="192.168.20.{0}".format(n) device = ConnectHandler(device_type='cisco_ios', ip=ip, username='test', password='test') output = device.send_command("show run | in hostname") output=output.split(" ") hostname=output[1] generatedconfig=template.replace("{rtr}",hostname) #step 2 #create different config files for each router ready to be pushed on routers. configfile=open(hostname+"_syslog_config.txt","w") configfile.write(generatedconfig) configfile.close() #step3 (Validation) #read files for each of the router (created as routername_syslog_config.txt) print ("Showing contents for generated config files....") for file in os.listdir('./'): if file.endswith(".txt"): if ("syslog_config" in file): hostname=file.split("_")[0] fileconfig=open(file) print ("\nShowing contents of "+hostname) print (fileconfig.read()) fileconfig.close() The output of running the preceding command is as follows: In a similar fashion to the previous example, the configuration is now generated. However, this time, instead of being pushed directly on routers, it is stored in different files with filenames based upon router names for all the routers that were provided in input. In each case, a .txt file is created (here is a sample filename that will be generated during execution of the script: rtr1_syslog_config.txt for the rtr1 router). As a final validation step, we read all the .txt files and print the generated configuration for each of the text files that has the naming convention containing syslog_config in the filename. There are times when we have a multi-vendor environment, and to manually create a customized configuration is a difficult task. Let us see an example in which we leverage a library (PySNMP) to fetch details regarding the given devices in the infrastructure using Simple Network Management Protocol (SNMP). For our test, we are using the SNMP community key mytest on the routers to fetch their model/version. The code to get the version and model of router, is as follows: #snmp_python.py from pysnmp.hlapi import * for n in range(1, 3): server_ip="192.168.20.{0}".format(n) errorIndication, errorStatus, errorIndex, varBinds = next( getCmd(SnmpEngine(), CommunityData('mytest', mpModel=0), UdpTransportTarget((server_ip, 161)), ContextData(), ObjectType(ObjectIdentity('SNMPv2-MIB', 'sysDescr', 0))) ) print ("\nFetching stats for...", server_ip) for varBind in varBinds: print (varBind[1]) The output of running the preceding command is as follows: As we see in this, the SNMP query was performed on a couple of routers (192.168.20.1 and 192.168.20.2). The SNMP query was performed using the standard Management Information Base (MIB), sysDescr. The return value of the routers against this MIB request is the make and model of the router and the current OS version it is running on. Using SNMP, we can fetch many vital statistics of the infrastructure and can generate configurations based upon the return values. This ensures that we have standard configurations even with a multi-vendor environment. As a sample, let us use the SNMP approach to determine the number of interfaces that a particular router has and, based upon the return values, we can dynamically generate a configuration irrespective of any number of interfaces available on the device. The code to fetch the available interfaces in a router is as follows: #snmp_python_interfacestats.py from pysnmp.entity.rfc3413.oneliner import cmdgen cmdGen = cmdgen.CommandGenerator() for n in range(1, 3): server_ip="192.168.20.{0}".format(n) print ("\nFetching stats for...", server_ip) errorIndication, errorStatus, errorIndex, varBindTable = cmdGen.bulkCmd( cmdgen.CommunityData('mytest'), cmdgen.UdpTransportTarget((server_ip, 161)), 0,25, '1.3.6.1.2.1.2.2.1.2' ) for varBindTableRow in varBindTable: for name, val in varBindTableRow: print('%s = Interface Name: %s' % (name.prettyPrint(), val.prettyPrint())) The output of running the preceding command is as follows: Using the snmpbulkwalk, we query for the interfaces on the router. The result from the query is a list that is parsed to fetch the SNMP MIB ID for the interfaces, along with the description of the interface. Multithreading A key focus area while performing operations on multiple devices is how quickly we can perform the actions. To put this into perspective, if each router takes around 10 seconds to log in, gather the output, and log out, and we have around 30 routers that we need to get this information from, we would need 10*30 = 300 seconds for the program to complete the execution. If we are looking for more advanced or complex calculations on each output, which might take up to a minute, then it will take 30 minutes for just 30 routers. This starts becoming very inefficient when our complexity and scalability grows. To help with this, we need to add parallelism to our programs.  Let us log in to each of the routers and fetch the show version using a parallel calling (or multithreading): #parallel_query.py from netmiko import ConnectHandler from datetime import datetime from threading import Thread startTime = datetime.now() threads = [] def checkparallel(ip): device = ConnectHandler(device_type='cisco_ios', ip=ip, username='test', password='test') output = device.send_command("show run | in hostname") output=output.split(" ") hostname=output[1] print ("\nHostname for IP %s is %s" % (ip,hostname)) for n in range(1, 5): ip="192.168.20.{0}".format(n) t = Thread(target=checkparallel, args= (ip,)) t.start() threads.append(t) #wait for all threads to completed for t in threads: t.join() print ("\nTotal execution time:") print(datetime.now() - startTime) The output of running the preceding command is as follows: The calling to the same set of routers being done in parallel takes approximately 8 seconds to fetch the results. Summary In this tutorial, we learned how to interact with Network devices through Python and got familiar with an extensively used library of Python (Netmiko) for network interactions. You also learned how to interact with multiple network devices using a simulated lab in GNS3 and got to know the device interaction through SNMP. Additionally, we also touched base on multithreading, which is a key component in scalability through various examples. To learn how to make your network robust by leveraging the power of Python, Ansible and other network automation tools, check out our book Practical Network Automation - Second Edition. AWS announces more flexibility its Certification Exams, drops its exam prerequisites Top 10 IT certifications for cloud and networking professionals in 2018 What matters on an engineering resume? Hacker Rank report says skills, not certifications