new

LICENSE

@@ -0,0 +1,353 @@
+Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright and
+certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+Considerations for licensors: Our public licenses are intended for use
+by those authorized to give the public permission to use material in
+ways otherwise restricted by copyright and certain other rights. Our
+licenses are irrevocable. Licensors should read and understand the terms
+and conditions of the license they choose before applying it. Licensors
+should also secure all rights necessary before applying our licenses so
+that the public can reuse the material as expected. Licensors should
+clearly mark any material not subject to the license. This includes
+other CC-licensed material, or material used under an exception or
+limitation to copyright. More considerations for licensors :
+wiki.creativecommons.org/Considerations_for_licensors
+
+Considerations for the public: By using one of our public licenses, a
+licensor grants the public permission to use the licensed material under
+specified terms and conditions. If the licensor's permission is not
+necessary for any reason–for example, because of any applicable
+exception or limitation to copyright–then that use is not regulated by
+the license. Our licenses grant only permissions under copyright and
+certain other rights that a licensor has authority to grant. Use of the
+licensed material may still be restricted for other reasons, including
+because others have copyright or other rights in the material. A
+licensor may make special requests, such as asking that all changes be
+marked or described. Although not required by our licenses, you are
+encouraged to respect those requests where reasonable. More
+considerations for the public :
+wiki.creativecommons.org/Considerations_for_licensees
+
+Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
+Public License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution-NonCommercial-ShareAlike 4.0 International Public License
+("Public License"). To the extent this Public License may be interpreted
+as a contract, You are granted the Licensed Rights in consideration of
+Your acceptance of these terms and conditions, and the Licensor grants
+You such rights in consideration of benefits the Licensor receives from
+making the Licensed Material available under these terms and conditions.
+
+Section 1 – Definitions.
+
+-   a. Adapted Material means material subject to Copyright and Similar
+    Rights that is derived from or based upon the Licensed Material and
+    in which the Licensed Material is translated, altered, arranged,
+    transformed, or otherwise modified in a manner requiring permission
+    under the Copyright and Similar Rights held by the Licensor. For
+    purposes of this Public License, where the Licensed Material is a
+    musical work, performance, or sound recording, Adapted Material is
+    always produced where the Licensed Material is synched in timed
+    relation with a moving image.
+-   b. Adapter's License means the license You apply to Your Copyright
+    and Similar Rights in Your contributions to Adapted Material in
+    accordance with the terms and conditions of this Public License.
+-   c. BY-NC-SA Compatible License means a license listed at
+    creativecommons.org/compatiblelicenses, approved by Creative Commons
+    as essentially the equivalent of this Public License.
+-   d. Copyright and Similar Rights means copyright and/or similar
+    rights closely related to copyright including, without limitation,
+    performance, broadcast, sound recording, and Sui Generis Database
+    Rights, without regard to how the rights are labeled or categorized.
+    For purposes of this Public License, the rights specified in Section
+    2(b)(1)-(2) are not Copyright and Similar Rights.
+-   e. Effective Technological Measures means those measures that, in
+    the absence of proper authority, may not be circumvented under laws
+    fulfilling obligations under Article 11 of the WIPO Copyright Treaty
+    adopted on December 20, 1996, and/or similar international
+    agreements.
+-   f. Exceptions and Limitations means fair use, fair dealing, and/or
+    any other exception or limitation to Copyright and Similar Rights
+    that applies to Your use of the Licensed Material.
+-   g. License Elements means the license attributes listed in the name
+    of a Creative Commons Public License. The License Elements of this
+    Public License are Attribution, NonCommercial, and ShareAlike.
+-   h. Licensed Material means the artistic or literary work, database,
+    or other material to which the Licensor applied this Public License.
+-   i. Licensed Rights means the rights granted to You subject to the
+    terms and conditions of this Public License, which are limited to
+    all Copyright and Similar Rights that apply to Your use of the
+    Licensed Material and that the Licensor has authority to license.
+-   j. Licensor means the individual(s) or entity(ies) granting rights
+    under this Public License.
+-   k. NonCommercial means not primarily intended for or directed
+    towards commercial advantage or monetary compensation. For purposes
+    of this Public License, the exchange of the Licensed Material for
+    other material subject to Copyright and Similar Rights by digital
+    file-sharing or similar means is NonCommercial provided there is no
+    payment of monetary compensation in connection with the exchange.
+-   l. Share means to provide material to the public by any means or
+    process that requires permission under the Licensed Rights, such as
+    reproduction, public display, public performance, distribution,
+    dissemination, communication, or importation, and to make material
+    available to the public including in ways that members of the public
+    may access the material from a place and at a time individually
+    chosen by them.
+-   m. Sui Generis Database Rights means rights other than copyright
+    resulting from Directive 96/9/EC of the European Parliament and of
+    the Council of 11 March 1996 on the legal protection of databases,
+    as amended and/or succeeded, as well as other essentially equivalent
+    rights anywhere in the world.
+-   n. You means the individual or entity exercising the Licensed Rights
+    under this Public License. Your has a corresponding meaning.
+
+Section 2 – Scope.
+
+-   a. License grant.
+    -   1. Subject to the terms and conditions of this Public License,
+        the Licensor hereby grants You a worldwide, royalty-free,
+        non-sublicensable, non-exclusive, irrevocable license to
+        exercise the Licensed Rights in the Licensed Material to:
+        -   A. reproduce and Share the Licensed Material, in whole or in
+            part, for NonCommercial purposes only; and
+        -   B. produce, reproduce, and Share Adapted Material for
+            NonCommercial purposes only.
+    -   2. Exceptions and Limitations. For the avoidance of doubt, where
+        Exceptions and Limitations apply to Your use, this Public
+        License does not apply, and You do not need to comply with its
+        terms and conditions.
+    -   3. Term. The term of this Public License is specified in Section
+        6(a).
+    -   4. Media and formats; technical modifications allowed. The
+        Licensor authorizes You to exercise the Licensed Rights in all
+        media and formats whether now known or hereafter created, and to
+        make technical modifications necessary to do so. The Licensor
+        waives and/or agrees not to assert any right or authority to
+        forbid You from making technical modifications necessary to
+        exercise the Licensed Rights, including technical modifications
+        necessary to circumvent Effective Technological Measures. For
+        purposes of this Public License, simply making modifications
+        authorized by this Section 2(a)(4) never produces Adapted
+        Material.
+    -   5. Downstream recipients.
+        -   A. Offer from the Licensor – Licensed Material. Every
+            recipient of the Licensed Material automatically receives an
+            offer from the Licensor to exercise the Licensed Rights
+            under the terms and conditions of this Public License.
+        -   B. Additional offer from the Licensor – Adapted Material.
+            Every recipient of Adapted Material from You automatically
+            receives an offer from the Licensor to exercise the Licensed
+            Rights in the Adapted Material under the conditions of the
+            Adapter's License You apply.
+        -   C. No downstream restrictions. You may not offer or impose
+            any additional or different terms or conditions on, or apply
+            any Effective Technological Measures to, the Licensed
+            Material if doing so restricts exercise of the Licensed
+            Rights by any recipient of the Licensed Material.
+    -   6. No endorsement. Nothing in this Public License constitutes or
+        may be construed as permission to assert or imply that You are,
+        or that Your use of the Licensed Material is, connected with, or
+        sponsored, endorsed, or granted official status by, the Licensor
+        or others designated to receive attribution as provided in
+        Section 3(a)(1)(A)(i).
+-   b. Other rights.
+    -   1. Moral rights, such as the right of integrity, are not
+        licensed under this Public License, nor are publicity, privacy,
+        and/or other similar personality rights; however, to the extent
+        possible, the Licensor waives and/or agrees not to assert any
+        such rights held by the Licensor to the limited extent necessary
+        to allow You to exercise the Licensed Rights, but not otherwise.
+    -   2. Patent and trademark rights are not licensed under this
+        Public License.
+    -   3. To the extent possible, the Licensor waives any right to
+        collect royalties from You for the exercise of the Licensed
+        Rights, whether directly or through a collecting society under
+        any voluntary or waivable statutory or compulsory licensing
+        scheme. In all other cases the Licensor expressly reserves any
+        right to collect such royalties, including when the Licensed
+        Material is used other than for NonCommercial purposes.
+
+Section 3 – License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+-   a. Attribution.
+    -   1. If You Share the Licensed Material (including in modified
+        form), You must:
+        -   A. retain the following if it is supplied by the Licensor
+            with the Licensed Material:
+            -   i. identification of the creator(s) of the Licensed
+                Material and any others designated to receive
+                attribution, in any reasonable manner requested by the
+                Licensor (including by pseudonym if designated);
+            -   ii. a copyright notice;
+            -   iii. a notice that refers to this Public License;
+            -   iv. a notice that refers to the disclaimer of
+                warranties;
+            -   v. a URI or hyperlink to the Licensed Material to the
+                extent reasonably practicable;
+
+        -   B. indicate if You modified the Licensed Material and retain
+            an indication of any previous modifications; and
+        -   C. indicate the Licensed Material is licensed under this
+            Public License, and include the text of, or the URI or
+            hyperlink to, this Public License.
+    -   2. You may satisfy the conditions in Section 3(a)(1) in any
+        reasonable manner based on the medium, means, and context in
+        which You Share the Licensed Material. For example, it may be
+        reasonable to satisfy the conditions by providing a URI or
+        hyperlink to a resource that includes the required information.
+    -   3. If requested by the Licensor, You must remove any of the
+        information required by Section 3(a)(1)(A) to the extent
+        reasonably practicable.
+-   b. ShareAlike.In addition to the conditions in Section 3(a), if You
+    Share Adapted Material You produce, the following conditions also
+    apply.
+    -   1. The Adapter's License You apply must be a Creative Commons
+        license with the same License Elements, this version or later,
+        or a BY-NC-SA Compatible License.
+    -   2. You must include the text of, or the URI or hyperlink to, the
+        Adapter's License You apply. You may satisfy this condition in
+        any reasonable manner based on the medium, means, and context in
+        which You Share Adapted Material.
+    -   3. You may not offer or impose any additional or different terms
+        or conditions on, or apply any Effective Technological Measures
+        to, Adapted Material that restrict exercise of the rights
+        granted under the Adapter's License You apply.
+
+Section 4 – Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that apply
+to Your use of the Licensed Material:
+
+-   a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+    to extract, reuse, reproduce, and Share all or a substantial portion
+    of the contents of the database for NonCommercial purposes only;
+-   b. if You include all or a substantial portion of the database
+    contents in a database in which You have Sui Generis Database
+    Rights, then the database in which You have Sui Generis Database
+    Rights (but not its individual contents) is Adapted Material,
+    including for purposes of Section 3(b); and
+-   c. You must comply with the conditions in Section 3(a) if You Share
+    all or a substantial portion of the contents of the database.
+    For the avoidance of doubt, this Section 4 supplements and does not
+    replace Your obligations under this Public License where the
+    Licensed Rights include other Copyright and Similar Rights.
+
+Section 5 – Disclaimer of Warranties and Limitation of Liability.
+
+-   a. Unless otherwise separately undertaken by the Licensor, to the
+    extent possible, the Licensor offers the Licensed Material as-is and
+    as-available, and makes no representations or warranties of any kind
+    concerning the Licensed Material, whether express, implied,
+    statutory, or other. This includes, without limitation, warranties
+    of title, merchantability, fitness for a particular purpose,
+    non-infringement, absence of latent or other defects, accuracy, or
+    the presence or absence of errors, whether or not known or
+    discoverable. Where disclaimers of warranties are not allowed in
+    full or in part, this disclaimer may not apply to You.
+-   b. To the extent possible, in no event will the Licensor be liable
+    to You on any legal theory (including, without limitation,
+    negligence) or otherwise for any direct, special, indirect,
+    incidental, consequential, punitive, exemplary, or other losses,
+    costs, expenses, or damages arising out of this Public License or
+    use of the Licensed Material, even if the Licensor has been advised
+    of the possibility of such losses, costs, expenses, or damages.
+    Where a limitation of liability is not allowed in full or in part,
+    this limitation may not apply to You.
+-   c. The disclaimer of warranties and limitation of liability provided
+    above shall be interpreted in a manner that, to the extent possible,
+    most closely approximates an absolute disclaimer and waiver of all
+    liability.
+
+Section 6 – Term and Termination.
+
+-   a. This Public License applies for the term of the Copyright and
+    Similar Rights licensed here. However, if You fail to comply with
+    this Public License, then Your rights under this Public License
+    terminate automatically.
+-   b. Where Your right to use the Licensed Material has terminated
+    under Section 6(a), it reinstates:
+
+    -   1. automatically as of the date the violation is cured, provided
+        it is cured within 30 days of Your discovery of the violation;
+        or
+    -   2. upon express reinstatement by the Licensor.
+
+    For the avoidance of doubt, this Section 6(b) does not affect any
+    right the Licensor may have to seek remedies for Your violations of
+    this Public License.
+
+-   c. For the avoidance of doubt, the Licensor may also offer the
+    Licensed Material under separate terms or conditions or stop
+    distributing the Licensed Material at any time; however, doing so
+    will not terminate this Public License.
+-   d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+    License.
+
+Section 7 – Other Terms and Conditions.
+
+-   a. The Licensor shall not be bound by any additional or different
+    terms or conditions communicated by You unless expressly agreed.
+-   b. Any arrangements, understandings, or agreements regarding the
+    Licensed Material not stated herein are separate from and
+    independent of the terms and conditions of this Public License.
+
+Section 8 – Interpretation.
+
+-   a. For the avoidance of doubt, this Public License does not, and
+    shall not be interpreted to, reduce, limit, restrict, or impose
+    conditions on any use of the Licensed Material that could lawfully
+    be made without permission under this Public License.
+-   b. To the extent possible, if any provision of this Public License
+    is deemed unenforceable, it shall be automatically reformed to the
+    minimum extent necessary to make it enforceable. If the provision
+    cannot be reformed, it shall be severed from this Public License
+    without affecting the enforceability of the remaining terms and
+    conditions.
+-   c. No term or condition of this Public License will be waived and no
+    failure to comply consented to unless expressly agreed to by the
+    Licensor.
+-   d. Nothing in this Public License constitutes or may be interpreted
+    as a limitation upon, or waiver of, any privileges and immunities
+    that apply to the Licensor or You, including from the legal
+    processes of any jurisdiction or authority.
+
+Creative Commons is not a party to its public licenses. Notwithstanding,
+Creative Commons may elect to apply one of its public licenses to
+material it publishes and in those instances will be considered the
+"Licensor." The text of the Creative Commons public licenses is
+dedicated to the public domain under the CC0 Public Domain Dedication.
+Except for the limited purpose of indicating that material is shared
+under a Creative Commons public license or as otherwise permitted by the
+Creative Commons policies published at creativecommons.org/policies,
+Creative Commons does not authorize the use of the trademark "Creative
+Commons" or any other trademark or logo of Creative Commons without its
+prior written consent including, without limitation, in connection with
+any unauthorized modifications to any of its public licenses or any
+other arrangements, understandings, or agreements concerning use of
+licensed material. For the avoidance of doubt, this paragraph does not
+form part of the public licenses.
+
+Creative Commons may be contacted at creativecommons.org.

README.md

@@ -1,3 +1,110 @@
----
-license: cc-by-nc-sa-4.0
----
+# Universal Activation Function
+
+Tensorflow and Pytorch source code for the paper 
+
+[Yuen, Brosnan, Minh Tu Hoang, Xiaodai Dong, and Tao Lu. "Universal activation function for machine learning." Scientific reports 11, no. 1 (2021): 1-11.](https://www.nature.com/articles/s41598-021-96723-8)
+
+
+# Getting the code
+
+Requires [Docker](https://docs.docker.com/get-docker/) 
+
+
+Use git to pull this repo
+```
+git clone https://github.com/SensorOrgNet/Universal_Activation_Function.git
+```
+
+
+# Running the Tensorflow 2 version
+
+
+Install CUDA 11.2 container
+```
+docker run --name UAF --gpus all  -v /home/username/UAF/:/workspace  -w /workspace    -it  nvcr.io/nvidia/cuda:11.2.0-cudnn8-devel-ubuntu20.04   bash
+```
+
+
+Install python
+```
+apt update
+apt install python3-pip
+```
+
+
+Install pytorch and pytorch geometric
+```
+pip3 install tensorflow==2.7.0
+```
+
+
+
+
+Run the MLP with UAF for MNIST dataset
+```
+cd   Universal_Activation_Function/tensorflow/
+python3   ./mnist_UAF.py 
+```
+
+
+
+
+
+
+# Running the Pytorch version
+
+
+Install CUDA 11.3 container
+```
+docker run --name UAF --gpus all  -v /home/username/UAF/:/workspace  -w /workspace    -it  nvcr.io/nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04   bash
+```
+
+Install python
+```
+apt update
+apt install python3-pip
+```
+
+Install pytorch and pytorch geometric
+```
+pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
+pip3 install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
+pip3 install torch-sparse -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
+pip3 install torch-cluster -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
+pip3 install torch-spline-conv -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
+pip3 install torch-geometric
+```
+
+
+
+
+Run the CNN with UAF for MNIST dataset
+```
+cd   Universal_Activation_Function/pytorch/
+python3   ./mnist_UAF.py 
+```
+
+
+
+
+
+
+
+Run the GCN2 with UAF for CORA dataset. The fold number is represented by the number at the end
+```
+cd   Universal_Activation_Function/pytorch/
+python3   ./gcn2_cora_UAF.py  0
+```
+
+
+
+
+
+Run the PNA with UAF for ZNC dataset. The fold number is represented by the number at the end
+```
+cd   Universal_Activation_Function/pytorch/
+python3   ./pna_UAF.py  0
+```
+
+
+

pytorch/gcn2_cora_UAF.py

@@ -0,0 +1,185 @@
+import os.path as osp
+import sys
+import time
+
+
+import torch
+from torch.nn import Linear
+import torch.nn.functional as F
+from torch_geometric.datasets import Planetoid
+import torch_geometric.transforms as T
+from torch_geometric.nn import GCN2Conv
+from torch_geometric.nn.conv.gcn_conv import gcn_norm
+import numpy as np
+
+train_pred = []
+train_act = []
+
+test_pred = []
+test_act = []
+
+fold = int(sys.argv[1])
+
+st = time.process_time()
+
+
+
+dataset = 'Cora'
+path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
+transform = T.Compose([T.NormalizeFeatures(), T.ToSparseTensor()])
+dataset = Planetoid(path, dataset, transform=transform)
+data = dataset[0]
+data.adj_t = gcn_norm(data.adj_t)  # Pre-process GCN normalization.
+
+
+
+ims = []
+
+
+
+
+
+
+class Net(torch.nn.Module):
+    def __init__(self, hidden_channels, num_layers, alpha, theta,
+                 shared_weights=True, dropout=0.0):
+        super(Net, self).__init__()
+
+        self.lins = torch.nn.ModuleList()
+        self.lins.append(Linear(dataset.num_features, hidden_channels))
+        self.lins.append(Linear(hidden_channels, dataset.num_classes))
+
+        self.convs = torch.nn.ModuleList()
+        for layer in range(num_layers):
+            self.convs.append(
+                GCN2Conv(hidden_channels, alpha, theta, layer + 1,
+                         shared_weights, normalize=False))
+
+        self.dropout = dropout
+
+        self.A = torch.nn.Parameter(torch.tensor(1.1, requires_grad=True))
+        self.B = torch.nn.Parameter(torch.tensor(-0.01, requires_grad=True))
+        self.C = torch.nn.Parameter(torch.tensor(1e-9, requires_grad=True))
+        self.D = torch.nn.Parameter(torch.tensor(-0.9, requires_grad=True))
+        self.E = torch.nn.Parameter(torch.tensor(0.00001, requires_grad=True))
+
+
+    def UAF(self, input):
+        ims.append(np.array([self.A.cpu().detach().item(),self.B.cpu().detach().item(),self.C.cpu().detach().item(),self.D.cpu().detach().item(),self.E.cpu().detach().item()]))
+        P1 = (self.A*(input+self.B)) + torch.clamp((self.C * torch.square(input)),-100.0,100.0)
+        P2 = (self.D*(input-self.B))
+
+        P3 = torch.nn.ReLU()(P1) + torch.log1p(torch.exp(-torch.abs(P1)))
+        P4 = torch.nn.ReLU()(P2) + torch.log1p(torch.exp(-torch.abs(P2)))
+        return P3 - P4  + self.E
+
+    def forward(self, x, adj_t):
+        x = F.dropout(x, self.dropout, training=self.training)
+        x = x_0 = self.UAF(self.lins[0](x))
+
+        for conv in self.convs:
+            x = F.dropout(x, self.dropout, training=self.training)
+            x = conv(x, x_0, adj_t)
+            x = self.UAF(x)
+
+        x = F.dropout(x, self.dropout, training=self.training)
+        x = self.lins[1](x)
+
+        return x.log_softmax(dim=-1)
+
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+model = Net(hidden_channels=64, num_layers=64, alpha=0.1, theta=0.5,
+            shared_weights=True, dropout=0.6).to(device)
+data = data.to(device)
+optimizer = torch.optim.Adam([
+    dict(params=model.convs.parameters(), weight_decay=0.01),
+    dict(params=model.lins.parameters(), weight_decay=5e-4)
+], lr=0.01)
+
+scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min',
+                                                       factor=0.7, patience=50,
+                                                       min_lr=0.00001)
+
+
+optimizer2 = torch.optim.Adam([
+    dict(params=model.A),
+    dict(params=model.B),
+    dict(params=model.C, weight_decay=1e5),
+    dict(params=model.D),
+    dict(params=model.E)
+], lr=0.005)
+scheduler2 = torch.optim.lr_scheduler.StepLR(optimizer2, step_size=240, gamma=1e-10)
+
+
+
+def train():
+    model.train()
+    optimizer2.zero_grad()
+    optimizer.zero_grad()
+    out = model(data.x, data.adj_t)
+    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
+
+    train_pred_temp = out[data.train_mask].cpu().detach().numpy()
+    train_act_temp = data.y[data.train_mask].cpu().detach().numpy()
+
+    train_pred.append(train_pred_temp)
+    train_act.append(train_act_temp)
+
+
+    loss.backward()
+    optimizer.step()
+    optimizer2.step()
+    return float(loss)
+
+
+@torch.no_grad()
+def test():
+    model.eval()
+    pred, accs = model(data.x, data.adj_t).argmax(dim=-1), []
+
+    test_pred_temp = pred[data.test_mask].cpu().detach().numpy()
+    test_act_temp = data.y[data.test_mask].cpu().detach().numpy()
+
+    test_pred.append(test_pred_temp)
+    test_act.append(test_act_temp)
+
+    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
+        accs.append(int((pred[mask] == data.y[mask]).sum()) / int(mask.sum()))
+    return accs
+
+
+best_val_acc = test_acc = 0
+for epoch in range(1, 1001):
+    loss = train()
+    train_acc, val_acc, tmp_test_acc = test()
+    if val_acc > best_val_acc:
+        best_val_acc = val_acc
+        test_acc = tmp_test_acc
+    lr = scheduler.optimizer.param_groups[0]['lr']
+    if (epoch == 241):
+        scheduler.optimizer.param_groups[0]['lr'] = 0.05
+    scheduler.step(-val_acc)
+    scheduler2.step()
+
+    print(f'Epoch: {epoch:04d}, Loss: {loss:.4f} Train: {train_acc:.4f}, '
+          f'lr: {lr:.7f}, Test: {tmp_test_acc:.4f}, '
+          f'Final Test: {test_acc:.4f}')
+
+
+
+elapsed_time = time.process_time() - st
+
+np.save("time_" + str(fold), np.array([elapsed_time]))
+
+
+np.save("train_pred_" + str(fold), train_pred)
+np.save("train_act_" + str(fold), train_act)
+
+
+np.save("test_pred_" + str(fold), test_pred)
+np.save("test_act_" + str(fold), test_act)
+
+
+ims = np.asarray(ims)
+np.save("ims_" + str(fold),ims)

pytorch/mnist_UAF.py

@@ -0,0 +1,167 @@
+from __future__ import print_function
+import argparse
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import datasets, transforms
+from torch.optim.lr_scheduler import StepLR
+
+
+
+class UAF(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.A = nn.Parameter(torch.tensor(30.0, requires_grad=True))
+        self.B = nn.Parameter(torch.tensor(0.0000012, requires_grad=True))
+        self.C = nn.Parameter(torch.tensor(0.0000001, requires_grad=True))
+        self.D = nn.Parameter(torch.tensor(29.0, requires_grad=True))
+        self.E = nn.Parameter(torch.tensor(0.00000102, requires_grad=True))
+
+    def forward(self, input):
+        P1 = (self.A*(input+self.B)) + (self.C * torch.square(input))
+        P2 = (self.D*(input-self.B))
+
+        P3 = nn.ReLU()(P1) + torch.log1p(torch.exp(-torch.abs(P1)))
+        P4 = nn.ReLU()(P2) + torch.log1p(torch.exp(-torch.abs(P2)))
+        return P3 - P4  + self.E
+
+
+
+
+
+
+
+class Net(nn.Module):
+    def __init__(self):
+        super(Net, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, 1)
+        self.conv2 = nn.Conv2d(32, 64, 3, 1)
+        self.dropout1 = nn.Dropout(0.25)
+        self.dropout2 = nn.Dropout(0.5)
+        self.fc1 = nn.Linear(9216, 128)
+        self.fc2 = nn.Linear(128, 10)
+        self.act0 = UAF()
+        self.act1 = UAF()
+        self.act2 = UAF()
+
+        
+        
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = self.act0(x)
+        x = self.conv2(x)
+        x = self.act1(x)
+        x = F.max_pool2d(x, 2)
+        x = self.dropout1(x)
+        x = torch.flatten(x, 1)
+        x = self.fc1(x)
+        x = self.act2(x)
+        x = self.dropout2(x)
+        x = self.fc2(x)
+        output = F.log_softmax(x, dim=1)
+        return output
+
+
+def train(args, model, device, train_loader, optimizer, epoch):
+    model.train()
+    for batch_idx, (data, target) in enumerate(train_loader):
+        data, target = data.to(device), target.to(device)
+        optimizer.zero_grad()
+        output = model(data)
+        loss = F.nll_loss(output, target)
+        loss.backward()
+        optimizer.step()
+        if batch_idx % args.log_interval == 0:
+            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
+                epoch, batch_idx * len(data), len(train_loader.dataset),
+                100. * batch_idx / len(train_loader), loss.item()))
+            if args.dry_run:
+                break
+
+
+def test(model, device, test_loader):
+    model.eval()
+    test_loss = 0
+    correct = 0
+    with torch.no_grad():
+        for data, target in test_loader:
+            data, target = data.to(device), target.to(device)
+            output = model(data)
+            test_loss += F.nll_loss(output, target, reduction='sum').item()  # sum up batch loss
+            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
+            correct += pred.eq(target.view_as(pred)).sum().item()
+
+    test_loss /= len(test_loader.dataset)
+
+    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
+        test_loss, correct, len(test_loader.dataset),
+        100. * correct / len(test_loader.dataset)))
+
+
+def main():
+    # Training settings
+    parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
+    parser.add_argument('--batch-size', type=int, default=64, metavar='N',
+                        help='input batch size for training (default: 64)')
+    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
+                        help='input batch size for testing (default: 1000)')
+    parser.add_argument('--epochs', type=int, default=14, metavar='N',
+                        help='number of epochs to train (default: 14)')
+    parser.add_argument('--lr', type=float, default=0.89, metavar='LR',
+                        help='learning rate (default: 0.9)')
+    parser.add_argument('--gamma', type=float, default=0.7, metavar='M',
+                        help='Learning rate step gamma (default: 0.7)')
+    parser.add_argument('--no-cuda', action='store_true', default=False,
+                        help='disables CUDA training')
+    parser.add_argument('--dry-run', action='store_true', default=False,
+                        help='quickly check a single pass')
+    parser.add_argument('--seed', type=int, default=1, metavar='S',
+                        help='random seed (default: 1)')
+    parser.add_argument('--log-interval', type=int, default=10, metavar='N',
+                        help='how many batches to wait before logging training status')
+    parser.add_argument('--save-model', action='store_true', default=False,
+                        help='For Saving the current Model')
+    args = parser.parse_args()
+    use_cuda = not args.no_cuda and torch.cuda.is_available()
+
+    torch.manual_seed(args.seed)
+
+    device = torch.device("cuda" if use_cuda else "cpu")
+
+    train_kwargs = {'batch_size': args.batch_size}
+    test_kwargs = {'batch_size': args.test_batch_size}
+    if use_cuda:
+        cuda_kwargs = {'num_workers': 1,
+                       'pin_memory': True,
+                       'shuffle': True}
+        train_kwargs.update(cuda_kwargs)
+        test_kwargs.update(cuda_kwargs)
+
+    transform=transforms.Compose([
+        transforms.ToTensor(),
+        transforms.Normalize((0.1307,), (0.3081,))
+        ])
+    dataset1 = datasets.MNIST('../data', train=True, download=True,
+                       transform=transform)
+    dataset2 = datasets.MNIST('../data', train=False,
+                       transform=transform)
+    train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
+    test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)
+
+    model = Net().to(device)
+    optimizer = optim.Adadelta(model.parameters(), lr=args.lr)
+
+    scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)
+    for epoch in range(1, args.epochs + 1):
+        train(args, model, device, train_loader, optimizer, epoch)
+        test(model, device, test_loader)
+        scheduler.step()
+
+    if args.save_model:
+        torch.save(model.state_dict(), "mnist_cnn.pt")
+
+
+if __name__ == '__main__':
+    main()

pytorch/pna_UAF.py

@@ -0,0 +1,248 @@
+import os.path as osp
+
+import torch
+import torch.nn.functional as F
+from torch.nn import ModuleList, Embedding
+from torch.nn import Sequential, ReLU, Linear
+from torch.optim.lr_scheduler import ReduceLROnPlateau
+from torch_geometric.utils import degree
+from torch_geometric.datasets import ZINC
+from torch_geometric.data import DataLoader
+from torch_geometric.nn import PNAConv, BatchNorm, global_add_pool
+
+
+
+
+
+
+import sys
+import time
+
+
+import numpy as np
+
+train_pred = []
+train_act = []
+
+test_pred = []
+test_act = []
+
+fold = int(sys.argv[1])
+
+st = time.process_time()
+
+
+
+
+
+
+
+path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ZINC')
+train_dataset = ZINC(path, subset=True, split='train')
+val_dataset = ZINC(path, subset=True, split='val')
+test_dataset = ZINC(path, subset=True, split='test')
+
+train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
+val_loader = DataLoader(val_dataset, batch_size=128)
+test_loader = DataLoader(test_dataset, batch_size=128)
+
+# Compute in-degree histogram over training data.
+deg = torch.zeros(5, dtype=torch.long)
+for data in train_dataset:
+    d = degree(data.edge_index[1], num_nodes=data.num_nodes, dtype=torch.long)
+    deg += torch.bincount(d, minlength=deg.numel())
+
+
+
+
+class UAF(torch.nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.A = torch.nn.Parameter(torch.tensor(1.1, requires_grad=True))
+        self.B = torch.nn.Parameter(torch.tensor(-0.01, requires_grad=True))
+        self.C = torch.nn.Parameter(torch.tensor(0.00001, requires_grad=True))
+        self.D = torch.nn.Parameter(torch.tensor(-0.9, requires_grad=True))
+        self.E = torch.nn.Parameter(torch.tensor(0.00001, requires_grad=True))
+
+        self.Softplus = torch.nn.Softplus()
+    def forward(self, input):
+
+        return self.Softplus((self.A*(input+self.B)) + (self.C * torch.square(input))) - self.Softplus((self.D*(input-self.B)))  + self.E
+
+
+
+class Net(torch.nn.Module):
+    def __init__(self):
+        super(Net, self).__init__()
+
+        self.node_emb = Embedding(21, 75)
+        self.edge_emb = Embedding(4, 50)
+
+        aggregators = ['mean', 'min', 'max', 'std']
+        scalers = ['identity', 'amplification', 'attenuation']
+
+        self.convs = ModuleList()
+        self.batch_norms = ModuleList()
+        for _ in range(4):
+            conv = PNAConv(in_channels=75, out_channels=75,
+                           aggregators=aggregators, scalers=scalers, deg=deg,
+                           edge_dim=50, towers=5, pre_layers=1, post_layers=1,
+                           divide_input=False)
+            self.convs.append(conv)
+            self.batch_norms.append(BatchNorm(75))
+
+        self.func =  UAF()
+
+
+        self.mlp = Sequential(Linear(75, 50), self.func, Linear(50, 25), self.func,
+                              Linear(25, 1))
+
+
+
+
+    def forward(self, x, edge_index, edge_attr, batch):
+        x = self.node_emb(x.squeeze())
+        edge_attr = self.edge_emb(edge_attr)
+
+        for conv, batch_norm in zip(self.convs, self.batch_norms):
+            x = self.func(batch_norm(conv(x, edge_index, edge_attr)))
+
+        x = global_add_pool(x, batch)
+        return self.mlp(x)
+
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+model = Net().to(device)
+#optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
+optimizer = torch.optim.Adam([
+    dict(params=model.convs.parameters()),
+    dict(params=model.batch_norms.parameters()),
+    dict(params=model.mlp.parameters())
+], lr=0.001)
+scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=20,
+                              min_lr=0.00001)
+
+
+
+optimizer2 = torch.optim.Adam([
+    dict(params=model.func.parameters())
+], lr=0.001)
+scheduler2 = torch.optim.lr_scheduler.StepLR(optimizer2, step_size=300, gamma=1e-10)
+
+
+
+
+
+
+def train(epoch):
+    model.train()
+
+
+    train_pred_temp = []
+    train_act_temp = []
+    first = True
+
+
+
+    total_loss = 0
+    for data in train_loader:
+        data = data.to(device)
+        optimizer.zero_grad()
+        optimizer2.zero_grad()
+        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
+        loss = (out.squeeze() - data.y).abs().mean()
+
+
+        pred = out.squeeze()
+        if (first):
+            train_pred_temp = pred.cpu().detach().numpy()
+            train_act_temp = data.y.cpu().detach().numpy()
+            first = False
+        else:
+            train_pred_temp = np.append(train_pred_temp, pred.cpu().detach().numpy())
+            train_act_temp = np.append(train_act_temp, data.y.cpu().detach().numpy())
+
+
+
+        loss.backward()
+        total_loss += loss.item() * data.num_graphs
+        optimizer.step()
+        optimizer2.step()
+
+
+    train_pred.append(train_pred_temp)
+    train_act.append(train_act_temp)
+    return total_loss / len(train_loader.dataset)
+
+
+@torch.no_grad()
+def test(loader):
+    model.eval()
+
+
+    test_pred_temp = []
+    test_act_temp = []
+    first = True
+
+    total_error = 0
+    for data in loader:
+        data = data.to(device)
+        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
+        total_error += (out.squeeze() - data.y).abs().sum().item()
+
+
+        pred = out.squeeze()
+        if (first):
+            test_pred_temp = pred.cpu().detach().numpy()
+            test_act_temp = data.y.cpu().detach().numpy()
+            first = False
+        else:
+            test_pred_temp = np.append(test_pred_temp, pred.cpu().detach().numpy())
+            test_act_temp = np.append(test_act_temp, data.y.cpu().detach().numpy())
+
+
+    test_pred.append(test_pred_temp)
+    test_act.append(test_act_temp)
+    return total_error / len(loader.dataset)
+
+
+
+@torch.no_grad()
+def test_val(loader):
+    model.eval()
+
+    total_error = 0
+    for data in loader:
+        data = data.to(device)
+        out = model(data.x, data.edge_index, data.edge_attr, data.batch)
+        total_error += (out.squeeze() - data.y).abs().sum().item()
+    return total_error / len(loader.dataset)
+
+
+
+
+for epoch in range(1, 601):
+    loss = train(epoch)
+    val_mae = test_val(val_loader)
+    test_mae = test(test_loader)
+    if (epoch == 302):
+        scheduler.optimizer.param_groups[0]['lr'] = 0.001
+    scheduler.step(val_mae)
+    scheduler2.step()
+    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_mae:.4f}, '
+          f'Test: {test_mae:.4f}')
+
+
+
+
+elapsed_time = time.process_time() - st
+
+np.save("time_" + str(fold), np.array([elapsed_time]))
+
+
+np.save("train_pred_" + str(fold), train_pred)
+np.save("train_act_" + str(fold), train_act)
+
+
+np.save("test_pred_" + str(fold), test_pred)
+np.save("test_act_" + str(fold), test_act)

tensorflow/CNNUAF_VGG16.py

@@ -0,0 +1,454 @@
+import tensorflow as tf
+import numpy as np
+import sys
+from sklearn.metrics import precision_recall_fscore_support
+import time
+
+
+def softplus(x):
+    return np.maximum(0, x) + np.log1p(np.exp(-np.abs(x)))
+
+
+def main():
+
+	neural_network_model_file = "./mlp.ckpt"
+
+
+	img_size = 32
+	num_channels = 3
+	output_size = 10
+	neuron_size = 4096
+	init_val = 1E-2
+	lr = 0.00001
+
+	size = int(sys.argv[1])
+
+	fold = int(sys.argv[2])
+
+
+	print ("Setting network up")
+	with tf.device("/device:GPU:0"):
+		is_training = tf.placeholder(tf.bool)
+		x = tf.placeholder("float32", [None, img_size, img_size, num_channels] , name="x")
+		y = tf.placeholder("float32", [None, output_size]  )
+		learning_rate = tf.placeholder("float32", None)
+
+		conv1_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 3, 64], mean=0, stddev=init_val))
+		conv2_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 64, 64], mean=0, stddev=init_val))
+
+		conv3_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 64, 128], mean=0, stddev=init_val))
+		conv4_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 128, 128], mean=0, stddev=init_val))
+
+		conv5_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 128, 256], mean=0, stddev=init_val))
+		conv6_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 256, 256], mean=0, stddev=init_val))
+		conv7_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 256, 256], mean=0, stddev=init_val))
+
+		conv8_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 256, 512], mean=0, stddev=init_val))
+		conv9_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 512, 512], mean=0, stddev=init_val))
+		conv10_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 512, 512], mean=0, stddev=init_val))
+
+
+		conv11_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 512, 512], mean=0, stddev=init_val))
+		conv12_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 512, 512], mean=0, stddev=init_val))
+		conv13_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 512, 512], mean=0, stddev=init_val))
+
+
+		cnnbias = {
+		'b1': tf.Variable(tf.random_normal([ 64 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b2': tf.Variable(tf.random_normal([ 64 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+
+		'b3': tf.Variable(tf.random_normal([ 128 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b4': tf.Variable(tf.random_normal([ 128 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+
+		'b5': tf.Variable(tf.random_normal([ 256 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b6': tf.Variable(tf.random_normal([ 256 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b7': tf.Variable(tf.random_normal([ 256 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+
+
+		'b8': tf.Variable(tf.random_normal([ 512 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b9': tf.Variable(tf.random_normal([ 512 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b10': tf.Variable(tf.random_normal([ 512 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+
+
+		'b11': tf.Variable(tf.random_normal([ 512 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b12': tf.Variable(tf.random_normal([ 512 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b13': tf.Variable(tf.random_normal([ 512 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32)
+
+		}
+
+
+
+		A = tf.Variable(tf.random_normal([1], mean=1.1,  stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+		B = tf.Variable(tf.random_normal([1], mean=-0.01, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+		C = tf.constant(0.0, dtype=tf.float32)
+		D = tf.Variable(tf.random_normal([1], mean=-0.9, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+		E = tf.Variable(tf.random_normal([1], mean=0.0, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+
+		three = tf.constant(3.0, dtype=tf.float32)
+		two = tf.constant(2.0, dtype=tf.float32)
+		one = tf.constant(1.0, dtype=tf.float32)
+		eps = tf.constant(1E-8, dtype=tf.float32)
+
+		def UAF(vv,UAF_A,UAF_B,UAF_C,UAF_D,UAF_E):
+			P1 = tf.multiply(UAF_A,(vv+UAF_B)) +tf.multiply(UAF_C,tf.pow(vv,two))
+			P2 = tf.multiply(UAF_D,(vv-UAF_B))
+
+			P3 = tf.nn.relu(P1) + tf.math.log1p(tf.exp(-tf.abs(P1)))
+			P4 = tf.nn.relu(P2) + tf.math.log1p(tf.exp(-tf.abs(P2)))
+			return P3 - P4  + UAF_E
+
+
+		conv1 = tf.nn.conv2d(x, conv1_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b1']
+		conv1_bn = tf.layers.batch_normalization(conv1,training = is_training)
+		conv1_act = UAF(conv1_bn,A,B,C,D,E)
+
+		conv2 = tf.nn.conv2d(conv1_act, conv2_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b2']
+		conv2_bn = tf.layers.batch_normalization(conv2,training = is_training)
+		conv2_act = UAF(conv2_bn,A,B,C,D,E)
+		conv2_pool = tf.nn.max_pool(conv2_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+
+
+
+		conv3 = tf.nn.conv2d(conv2_pool, conv3_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b3']
+		conv3_bn = tf.layers.batch_normalization(conv3,training = is_training)
+		conv3_act = UAF(conv3_bn,A,B,C,D,E)
+
+		conv4 = tf.nn.conv2d(conv3_act, conv4_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b4']
+		conv4_bn = tf.layers.batch_normalization(conv4,training = is_training)
+		conv4_act = UAF(conv4_bn,A,B,C,D,E)
+		conv4_pool = tf.nn.max_pool(conv4_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+
+
+
+
+		conv5 = tf.nn.conv2d(conv4_pool, conv5_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b5']
+		conv5_bn = tf.layers.batch_normalization(conv5,training = is_training)
+		conv5_act = UAF(conv5_bn,A,B,C,D,E)
+
+		conv6 = tf.nn.conv2d(conv5_act, conv6_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b6']
+		conv6_bn = tf.layers.batch_normalization(conv6,training = is_training)
+		conv6_act = UAF(conv6_bn,A,B,C,D,E)
+
+		conv7 = tf.nn.conv2d(conv6_act, conv7_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b7']
+		conv7_bn = tf.layers.batch_normalization(conv7,training = is_training)
+		conv7_act = UAF(conv7_bn,A,B,C,D,E)
+
+		conv7_pool = tf.nn.max_pool(conv7_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+
+
+
+
+		conv8 = tf.nn.conv2d(conv7_pool, conv8_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b8']
+		conv8_bn = tf.layers.batch_normalization(conv8,training = is_training)
+		conv8_act = UAF(conv8_bn,A,B,C,D,E)
+
+		conv9 = tf.nn.conv2d(conv8_act, conv9_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b9']
+		conv9_bn = tf.layers.batch_normalization(conv9,training = is_training)
+		conv9_act = UAF(conv9_bn,A,B,C,D,E)
+
+		conv10 = tf.nn.conv2d(conv9_act, conv10_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b10']
+		conv10_bn = tf.layers.batch_normalization(conv10,training = is_training)
+		conv10_act = UAF(conv10_bn,A,B,C,D,E)
+
+		conv10_pool = tf.nn.max_pool(conv10_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+
+
+
+		conv11 = tf.nn.conv2d(conv10_pool, conv11_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b11']
+		conv11_bn = tf.layers.batch_normalization(conv11,training = is_training)
+		conv11_act = UAF(conv11_bn,A,B,C,D,E)
+
+		conv12 = tf.nn.conv2d(conv11_act, conv12_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b12']
+		conv12_bn = tf.layers.batch_normalization(conv12,training = is_training)
+		conv12_act = UAF(conv12_bn,A,B,C,D,E)
+
+		conv13 = tf.nn.conv2d(conv12_act, conv13_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b13']
+		conv13_bn = tf.layers.batch_normalization(conv13,training = is_training)
+		conv13_act = UAF(conv13_bn,A,B,C,D,E)
+
+
+		# 9
+		flat = tf.contrib.layers.flatten(conv13_act)
+
+
+		weights = {
+		'w1': tf.Variable(tf.random_normal([ 2048, neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'w2': tf.Variable(tf.random_normal([ neuron_size, neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'w3': tf.Variable(tf.random_normal([ neuron_size, output_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32)
+		}
+
+		bias = {
+		'b1': tf.Variable(tf.random_normal([ neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b2': tf.Variable(tf.random_normal([ neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b3': tf.Variable(tf.random_normal([ output_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32)
+		}
+
+
+		# 10
+		Y1 = tf.layers.batch_normalization(tf.matmul( flat , weights['w1'] )+bias['b1'],training = is_training)
+		Y1act = UAF(Y1,A,B,C,D,E)
+
+		Y2 = tf.layers.batch_normalization(tf.matmul( Y1act , weights['w2'] )+bias['b2'],training = is_training)
+		Y2act = UAF(Y2,A,B,C,D,E)
+
+		# 12
+		yhat = UAF(tf.matmul( Y2act , weights['w3'] )+bias['b3'],A,B,C,D,E)
+
+
+		train_vars = [conv1_filter
+		, conv2_filter
+		, conv3_filter
+		, conv4_filter
+		, conv5_filter
+		, conv6_filter
+		, conv7_filter
+		, conv8_filter
+		, conv9_filter
+		, conv10_filter
+		, conv11_filter
+		, conv12_filter
+		, conv13_filter
+
+
+
+		, cnnbias['b1']
+		, cnnbias['b2']
+		, cnnbias['b3']
+		, cnnbias['b4']
+		, cnnbias['b5']
+		, cnnbias['b6']
+		, cnnbias['b7']
+		, cnnbias['b8']
+		, cnnbias['b9']
+		, cnnbias['b10']
+		, cnnbias['b11']
+		, cnnbias['b12']
+		, cnnbias['b13']
+
+
+		,weights['w1']
+		,weights['w2']
+		,weights['w3']
+
+		,bias['b1']
+		,bias['b2']
+		,bias['b3']
+		]
+
+		update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
+		update_ops2 = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
+
+
+
+		#ta = three*tf.math.sigmoid(tf.Variable(tf.random_normal([1], mean=0.5, stddev=0.000000001, dtype=tf.float32), dtype=tf.float32))
+		#vc = tf.Variable(tf.random_normal([1], mean=0.2, stddev=0.000000000001, dtype=tf.float32), dtype=tf.float32)
+		#tc = tf.nn.relu(vc) + tf.math.log1p(tf.exp(-tf.abs(vc))) + eps
+
+		#loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = yhat, labels = y))
+		loss = tf.math.sqrt(tf.reduce_mean(  tf.square(tf.subtract(y, yhat))  ) )
+
+
+
+
+		train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
+		train_op2 = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss, var_list=train_vars)
+		train_op = tf.group([train_op, update_ops])
+		train_op2 = tf.group([train_op2, update_ops2])
+
+
+
+
+		tf.set_random_seed(4123)
+
+		with tf.device("/device:CPU:0"):
+			saver = tf.train.Saver()
+
+		with tf.Session() as sess:
+			sess.run(tf.global_variables_initializer())
+
+			#tf.train.write_graph(sess.graph_def, '.', 'cnn_lstm.pbtxt')
+
+			#saver.restore(sess, neural_network_model_file)
+			ims = []
+			testloss = []
+			trainloss = []
+
+			prev = 10000
+			gl = 100
+			thres = 1E-10
+			epoch = 0
+			c = 1000
+
+
+			resetvar = 0
+			test_img = np.load("test_img.npy").astype("float32")
+			test_label = np.load("test_label.npy").astype("float32")
+
+			#for epoch in range(4000):
+			st = time.process_time()
+			while(1==1):
+				train_img = np.load("train_img"+str(epoch%size)+".npy").astype("float32")
+				train_label = np.load("train_label"+str(epoch%size)+".npy").astype("float32")
+
+
+
+				feed_dict = {y: train_label,
+				learning_rate: lr,
+				is_training: True,
+				x: train_img}
+
+
+				if (gl < 0.08):
+					lr = 0.000001
+				else:
+					lr = 0.00001
+
+				if (resetvar == 0):
+					_, gl= sess.run([ train_op, loss],
+					feed_dict=feed_dict)
+					if (gl < 0.01):
+						resetvar = 1
+						UA,UB,UD,UE= sess.run([ A,B,D,E],
+						feed_dict=feed_dict)
+						sess.run(tf.global_variables_initializer())
+						A.load(UA)
+						B.load(UB)
+						D.load(UD)
+						E.load(UE)
+				elif (resetvar == 1):
+					if (gl < 0.01):
+						resetvar = 2
+					_, gl= sess.run([ train_op2, loss],
+					feed_dict=feed_dict)
+				elif (resetvar == 2):
+					_, gl= sess.run([ train_op, loss],
+					feed_dict=feed_dict)
+
+
+				if (epoch % 100) == 0:
+					feed_dict = {is_training: False,
+					x: test_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(test_label,axis=1).astype("int32")
+					_,_,f1,_ = precision_recall_fscore_support(y_true, y_pred, average='macro')
+					testloss.append(f1)
+					print(precision_recall_fscore_support(y_true, y_pred, average='macro'))
+
+
+					feed_dict = {
+					is_training: False,
+					x: train_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(train_label,axis=1).astype("int32")
+					_,_,f1,_ = precision_recall_fscore_support(y_true, y_pred, average='macro')
+					trainloss.append(f1)
+
+					#UA,UB,UC,UD,UE= sess.run([ A,B,C,D,E],
+					#feed_dict=feed_dict)
+
+					UA,UB,UD,UE= sess.run([ A,B,D,E],
+					feed_dict=feed_dict)
+
+					ims.append(np.array([UA,UB,0.0,UD,UE]))
+
+
+
+
+
+
+                #loss_val.append([gl,dl])
+				print(epoch)
+				epoch = epoch + 1
+				print(gl)
+				prev = c
+				c = gl
+
+
+				if (epoch > 12000):
+					elapsed_time = time.process_time() - st
+
+					data_dir = "time_"+str(size)+ "_" + str(fold) + ".npy"
+					np.save(data_dir, np.array([elapsed_time, epoch]))
+
+
+					feed_dict = {
+					is_training: False,
+					x: test_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(test_label,axis=1).astype("int32")
+
+					data_dir = "test_pred_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_pred)
+
+					data_dir = "test_true_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_true)
+
+
+
+
+
+					feed_dict = {
+					is_training: False,
+					x: train_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(train_label,axis=1).astype("int32")
+
+					data_dir = "train_pred_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_pred)
+
+					data_dir = "train_true_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_true)
+
+
+					trainloss = np.array(trainloss)
+					np.save("trainloss"+str(size) + "_" + str(fold) + ".npy", trainloss)
+
+					testloss = np.array(testloss)
+					np.save("testloss"+str(size) + "_" + str(fold) + ".npy", testloss)
+
+					ims = np.asarray(ims)
+					np.save("ims"+str(size) + "_" + str(fold) + ".npy", ims)
+
+					break
+
+
+
+
+			#ani = animation.ArtistAnimation(fig, ims, interval=100, blit=True, repeat_delay=2000)
+			#ani.save('animation.gif', writer='imagemagick', fps=10)
+
+			saver.save(sess, neural_network_model_file)
+
+
+
+
+
+
+
+
+
+
+if __name__ == '__main__':
+    main()

tensorflow/CNNUAF_VGGBC.py

@@ -0,0 +1,383 @@
+import tensorflow as tf
+import numpy as np
+import sys
+from sklearn.metrics import precision_recall_fscore_support
+import time
+
+
+def softplus(x):
+    return np.maximum(0, x) + np.log1p(np.exp(-np.abs(x)))
+
+
+def main():
+
+	neural_network_model_file = "./mlp.ckpt"
+
+
+	img_size = 32
+	num_channels = 3
+	output_size = 10
+	neuron_size = 1024
+	init_val = 1E-2
+	lr = 0.0007
+
+	size = int(sys.argv[1])
+
+	fold = int(sys.argv[2])
+
+	input_keep_prob = 0.8
+
+
+	print ("Setting network up")
+	with tf.device("/device:GPU:0"):
+		is_training = tf.placeholder(tf.bool)
+		keep_prob = tf.placeholder(tf.float32, name='keep_prob')
+		x = tf.placeholder("float32", [None, img_size, img_size, num_channels] , name="x")
+		y = tf.placeholder("float32", [None, output_size]  )
+		learning_rate = tf.placeholder("float32", None)
+
+		conv1_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 3, 64], mean=0, stddev=init_val))
+		conv2_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 64, 64], mean=0, stddev=init_val))
+
+		conv3_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 64, 128], mean=0, stddev=init_val))
+		conv4_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 128, 128], mean=0, stddev=init_val))
+
+		conv5_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 128, 256], mean=0, stddev=init_val))
+		conv6_filter = tf.Variable(tf.truncated_normal(shape=[3, 3, 256, 256], mean=0, stddev=init_val))
+
+
+		cnnbias = {
+		'b1': tf.Variable(tf.random_normal([ 64 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b2': tf.Variable(tf.random_normal([ 64 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b3': tf.Variable(tf.random_normal([ 128 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b4': tf.Variable(tf.random_normal([ 128 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b5': tf.Variable(tf.random_normal([ 256 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b6': tf.Variable(tf.random_normal([ 256 ], stddev=init_val, dtype=tf.float32), dtype=tf.float32)
+		}
+
+
+
+		A = tf.Variable(tf.random_normal([1], mean=1.1,  stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+		B = tf.Variable(tf.random_normal([1], mean=-0.01, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+		C = tf.constant(0.0, dtype=tf.float32)
+		D = tf.Variable(tf.random_normal([1], mean=-0.9, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+		E = tf.Variable(tf.random_normal([1], mean=0.0, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+
+		three = tf.constant(3.0, dtype=tf.float32)
+		two = tf.constant(2.0, dtype=tf.float32)
+		one = tf.constant(1.0, dtype=tf.float32)
+		eps = tf.constant(1E-8, dtype=tf.float32)
+
+		def UAF(vv,UAF_A,UAF_B,UAF_C,UAF_D,UAF_E):
+			P1 = tf.multiply(UAF_A,(vv+UAF_B)) +tf.multiply(UAF_C,tf.pow(vv,two))
+			P2 = tf.multiply(UAF_D,(vv-UAF_B))
+
+			P3 = tf.nn.relu(P1) + tf.math.log1p(tf.exp(-tf.abs(P1)))
+			P4 = tf.nn.relu(P2) + tf.math.log1p(tf.exp(-tf.abs(P2)))
+			return P3 - P4  + UAF_E
+
+
+		conv1 = tf.nn.conv2d(x, conv1_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b1']
+		conv1_bn = tf.layers.batch_normalization(conv1,training = is_training)
+		conv1_act = UAF(conv1_bn,A,B,C,D,E)
+
+		conv2 = tf.nn.conv2d(conv1_act, conv2_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b2']
+		conv2_bn = tf.layers.batch_normalization(conv2,training = is_training)
+		conv2_act = UAF(conv2_bn,A,B,C,D,E)
+		conv2_pool = tf.nn.max_pool(conv2_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+		conv2_drop = tf.nn.dropout(conv2_pool, keep_prob)
+
+
+		conv3 = tf.nn.conv2d(conv2_drop, conv3_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b3']
+		conv3_bn = tf.layers.batch_normalization(conv3,training = is_training)
+		conv3_act = UAF(conv3_bn,A,B,C,D,E)
+
+		conv4 = tf.nn.conv2d(conv3_act, conv4_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b4']
+		conv4_bn = tf.layers.batch_normalization(conv4,training = is_training)
+		conv4_act = UAF(conv4_bn,A,B,C,D,E)
+		conv4_pool = tf.nn.max_pool(conv4_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+
+		conv4_drop = tf.nn.dropout(conv4_pool, keep_prob)
+
+
+
+		conv5 = tf.nn.conv2d(conv4_drop, conv5_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b5']
+		conv5_bn = tf.layers.batch_normalization(conv5,training = is_training)
+		conv5_act = UAF(conv5_bn,A,B,C,D,E)
+
+		conv6 = tf.nn.conv2d(conv5_act, conv6_filter, strides=[1,1,1,1], padding='SAME') + cnnbias['b6']
+		conv6_bn = tf.layers.batch_normalization(conv6,training = is_training)
+		conv6_act = UAF(conv6_bn,A,B,C,D,E)
+		conv6_pool = tf.nn.max_pool(conv6_act, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
+
+		conv6_drop = tf.nn.dropout(conv6_pool, keep_prob)
+
+
+		# 9
+		flat = tf.contrib.layers.flatten(conv6_drop)
+
+
+		weights = {
+		'w1': tf.Variable(tf.random_normal([ 4096, neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'w2': tf.Variable(tf.random_normal([ neuron_size, neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'w3': tf.Variable(tf.random_normal([ neuron_size, output_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32)
+		}
+
+		bias = {
+		'b1': tf.Variable(tf.random_normal([ neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b2': tf.Variable(tf.random_normal([ neuron_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32),
+		'b3': tf.Variable(tf.random_normal([ output_size ], stddev=init_val, dtype=tf.float32), dtype=tf.float32)
+		}
+
+
+		# 10
+		Y1 = tf.layers.batch_normalization(tf.matmul( flat , weights['w1'] )+bias['b1'],training = is_training)
+		Y1_drop = tf.nn.dropout(Y1, keep_prob)
+		Y1act = UAF(Y1_drop,A,B,C,D,E)
+
+
+		Y2 = tf.layers.batch_normalization(tf.matmul( Y1act , weights['w2'] )+bias['b2'],training = is_training)
+		Y2_drop = tf.nn.dropout(Y2, keep_prob)
+		Y2act = UAF(Y2_drop,A,B,C,D,E)
+
+		# 12
+		yhat = UAF(tf.matmul( Y2act , weights['w3'] )+bias['b3'],A,B,C,D,E)
+
+
+		train_vars = [conv1_filter
+		, conv2_filter
+		, conv3_filter
+		, conv4_filter
+		, conv5_filter
+		, conv6_filter
+
+		, cnnbias['b1']
+		, cnnbias['b2']
+		, cnnbias['b3']
+		, cnnbias['b4']
+		, cnnbias['b5']
+		, cnnbias['b6']
+
+		,weights['w1']
+		,weights['w2']
+		,weights['w3']
+
+		,bias['b1']
+		,bias['b2']
+		,bias['b3']
+		]
+
+		update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
+		update_ops2 = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
+
+
+
+		#ta = three*tf.math.sigmoid(tf.Variable(tf.random_normal([1], mean=0.5, stddev=0.000000001, dtype=tf.float32), dtype=tf.float32))
+		#vc = tf.Variable(tf.random_normal([1], mean=0.2, stddev=0.000000000001, dtype=tf.float32), dtype=tf.float32)
+		#tc = tf.nn.relu(vc) + tf.math.log1p(tf.exp(-tf.abs(vc))) + eps
+
+		#loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = yhat, labels = y))
+		loss = tf.math.sqrt(tf.reduce_mean(  tf.square(tf.subtract(y, yhat))  ) )
+
+
+
+
+		train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
+		train_op2 = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss, var_list=train_vars)
+		train_op = tf.group([train_op, update_ops])
+		train_op2 = tf.group([train_op2, update_ops2])
+
+
+
+
+		tf.set_random_seed(4123)
+
+		with tf.device("/device:CPU:0"):
+			saver = tf.train.Saver()
+
+		with tf.Session() as sess:
+			sess.run(tf.global_variables_initializer())
+
+			#tf.train.write_graph(sess.graph_def, '.', 'cnn_lstm.pbtxt')
+
+			#saver.restore(sess, neural_network_model_file)
+			ims = []
+			testloss = []
+			trainloss = []
+			stats = []
+
+			prev = 10000
+			gl = 100
+			thres = 1E-10
+			epoch = 0
+			c = 1000
+
+
+			resetvar = 0
+			test_img = np.load("test_img.npy").astype("float32")
+			test_label = np.load("test_label.npy").astype("float32")
+
+			#for epoch in range(4000):
+			st = time.process_time()
+			while(1==1):
+				train_img = np.load("train_img"+str(epoch%size)+".npy").astype("float32")
+				train_label = np.load("train_label"+str(epoch%size)+".npy").astype("float32")
+
+
+
+				feed_dict = {y: train_label,
+				learning_rate: lr,
+				is_training: True,
+				keep_prob: input_keep_prob,
+				x: train_img}
+
+
+				if (gl < 0.04):
+					lr = 0.00004
+				else:
+					lr = 0.003
+
+
+				if (gl < 0.14):
+					_, gl= sess.run([ train_op2, loss],
+					feed_dict=feed_dict)
+				else:
+					_, gl= sess.run([ train_op, loss],
+					feed_dict=feed_dict)
+
+
+
+				if (epoch % 100) == 0:
+					feed_dict = {
+					is_training: False,
+					keep_prob: 1.0,
+					x: test_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(test_label,axis=1).astype("int32")
+					pr,re,f1,_ = precision_recall_fscore_support(y_true, y_pred, average='macro')
+					testloss.append(f1)
+					print(precision_recall_fscore_support(y_true, y_pred, average='macro'))
+					stats.append(np.array([pr,re,f1]))
+
+					feed_dict = {
+					is_training: False,
+					keep_prob: 1.0,
+					x: train_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(train_label,axis=1).astype("int32")
+					_,_,f1,_ = precision_recall_fscore_support(y_true, y_pred, average='macro')
+					trainloss.append(f1)
+
+					#UA,UB,UC,UD,UE= sess.run([ A,B,C,D,E],
+					#feed_dict=feed_dict)
+
+					UA,UB,UD,UE= sess.run([ A,B,D,E],
+					feed_dict=feed_dict)
+
+					ims.append(np.array([UA,UB,0.0,UD,UE]))
+
+
+
+
+
+
+                #loss_val.append([gl,dl])
+				print(epoch)
+				epoch = epoch + 1
+				print(gl)
+				prev = c
+				c = gl
+
+
+				if (epoch > 10000):
+					elapsed_time = time.process_time() - st
+
+					data_dir = "time_"+str(size)+ "_" + str(fold) + ".npy"
+					np.save(data_dir, np.array([elapsed_time, epoch]))
+
+
+					feed_dict = {
+					is_training: False,
+					keep_prob: 1.0,
+					x: test_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(test_label,axis=1).astype("int32")
+
+					data_dir = "test_pred_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_pred)
+
+					data_dir = "test_true_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_true)
+
+
+
+
+
+					feed_dict = {
+					is_training: False,
+					keep_prob: 1.0,
+					x: train_img}
+
+					output_y = sess.run(yhat,
+	    			feed_dict=feed_dict)
+
+
+					y_pred = np.argmax(output_y,axis=1).astype("int32")
+					y_true = np.argmax(train_label,axis=1).astype("int32")
+
+					data_dir = "train_pred_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_pred)
+
+					data_dir = "train_true_"+str(size) + "_" + str(fold) + ".npy"
+					np.save(data_dir, y_true)
+
+
+					trainloss = np.array(trainloss)
+					np.save("trainloss"+str(size) + "_" + str(fold) + ".npy", trainloss)
+
+					testloss = np.array(testloss)
+					np.save("testloss"+str(size) + "_" + str(fold) + ".npy", testloss)
+
+					ims = np.asarray(ims)
+					np.save("ims"+str(size) + "_" + str(fold) + ".npy", ims)
+
+					stats = np.asarray(stats)
+					np.save("stats"+str(size) + "_" + str(fold) + ".npy", stats)
+
+
+					break
+
+
+
+
+			#ani = animation.ArtistAnimation(fig, ims, interval=100, blit=True, repeat_delay=2000)
+			#ani.save('animation.gif', writer='imagemagick', fps=10)
+
+			saver.save(sess, neural_network_model_file)
+
+
+
+
+
+
+
+
+
+
+if __name__ == '__main__':
+    main()

tensorflow/DDPG_UAF.py

@@ -0,0 +1,437 @@
+import tensorflow as tf
+import numpy as np
+import gym
+import os
+import shutil
+import sys
+import time
+
+
+np.random.seed(1)
+tf.set_random_seed(1)
+
+
+fold = int(sys.argv[1])
+
+st = time.process_time()
+
+MAX_EPISODES = 1500
+LR_A = 0.0005  # learning rate for actor
+LR_C = 0.0005  # learning rate for critic
+GAMMA = 0.999  # reward discount
+REPLACE_ITER_A = 1700
+REPLACE_ITER_C = 1500
+MEMORY_CAPACITY = 200000
+BATCH_SIZE = 32
+DISPLAY_THRESHOLD = 100  # display until the running reward > 100
+DATA_PATH = './data'
+LOAD_MODEL = False
+SAVE_MODEL_ITER = 100000
+RENDER = False
+OUTPUT_GRAPH = False
+ENV_NAME = 'BipedalWalker-v2'
+
+GLOBAL_STEP = tf.Variable(0, trainable=False)
+INCREASE_GS = GLOBAL_STEP.assign(tf.add(GLOBAL_STEP, 1))
+LR_A = tf.train.exponential_decay(LR_A, GLOBAL_STEP, 10000, .97, staircase=True)
+LR_C = tf.train.exponential_decay(LR_C, GLOBAL_STEP, 10000, .97, staircase=True)
+END_POINT = (200 - 10) * (14/30)    # from game
+
+env = gym.make(ENV_NAME)
+env.seed(1)
+
+STATE_DIM = env.observation_space.shape[0]  # 24
+ACTION_DIM = env.action_space.shape[0]  # 4
+ACTION_BOUND = env.action_space.high    # [1, 1, 1, 1]
+
+# all placeholder for tf
+with tf.name_scope('S'):
+    S = tf.placeholder(tf.float32, shape=[None, STATE_DIM], name='s')
+with tf.name_scope('R'):
+    R = tf.placeholder(tf.float32, [None, 1], name='r')
+with tf.name_scope('S_'):
+    S_ = tf.placeholder(tf.float32, shape=[None, STATE_DIM], name='s_')
+
+###############################  Actor  ####################################
+
+class Actor(object):
+    def __init__(self, sess, action_dim, action_bound, learning_rate, t_replace_iter):
+        self.sess = sess
+        self.a_dim = action_dim
+        self.action_bound = action_bound
+        self.lr = learning_rate
+        self.t_replace_iter = t_replace_iter
+        self.t_replace_counter = 0
+
+        with tf.variable_scope('Actor'):
+            # input s, output a
+            self.a = self._build_net(S, scope='eval_net', trainable=True)
+
+            # input s_, output a, get a_ for critic
+            self.a_ = self._build_net(S_, scope='target_net', trainable=False)
+
+        self.e_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Actor/eval_net')
+        self.t_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Actor/target_net')
+
+    def _build_net(self, s, scope, trainable):
+        with tf.variable_scope(scope):
+            init_w = tf.random_normal_initializer(0., 0.01)
+            init_b = tf.constant_initializer(0.01)
+
+            two = tf.constant(2.0, dtype=tf.float32)
+            def UAF(x):
+                UAF_A = tf.Variable(tf.random_normal([1], mean=1.1,  stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+                UAF_B = tf.Variable(tf.random_normal([1], mean=-0.01, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+                UAF_C = tf.Variable(tf.random_normal([1], mean=-0.1, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+                UAF_D = tf.Variable(tf.random_normal([1], mean=-0.9, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+                UAF_E = tf.Variable(tf.random_normal([1], mean=0.01, stddev=0.0, dtype=tf.float32), dtype=tf.float32)
+
+                P1 = tf.multiply(UAF_A,(x+UAF_B)) +tf.multiply(UAF_C,tf.pow(x,two))
+                P2 = tf.multiply(UAF_D,(x-UAF_B))
+
+                P3 = tf.nn.relu(P1) + tf.math.log1p(tf.exp(-tf.abs(P1)))
+                P4 = tf.nn.relu(P2) + tf.math.log1p(tf.exp(-tf.abs(P2)))
+                return P3 - P4  + UAF_E
+
+
+            net = tf.layers.dense(s, 500, activation=UAF,
+                                  kernel_initializer=init_w, bias_initializer=init_b, name='l1', trainable=trainable)
+            net = tf.layers.dense(net, 200, activation=UAF,
+                                  kernel_initializer=init_w, bias_initializer=init_b, name='l2', trainable=trainable)
+
+            with tf.variable_scope('a'):
+                actions = tf.layers.dense(net, self.a_dim, activation=tf.nn.tanh, kernel_initializer=init_w,
+                                          bias_initializer=init_b, name='a', trainable=trainable)
+                scaled_a = tf.multiply(actions, self.action_bound, name='scaled_a')  # Scale output to -action_bound to action_bound
+        return scaled_a
+
+    def learn(self, s):  # batch update
+        self.sess.run(self.train_op, feed_dict={S: s})
+        if self.t_replace_counter % self.t_replace_iter == 0:
+            self.sess.run([tf.assign(t, e) for t, e in zip(self.t_params, self.e_params)])
+        self.t_replace_counter += 1
+
+    def choose_action(self, s):
+        s = s[np.newaxis, :]    # single state
+        return self.sess.run(self.a, feed_dict={S: s})[0]  # single action
+
+    def add_grad_to_graph(self, a_grads):
+        with tf.variable_scope('policy_grads'):
+            # ys = policy;
+            # xs = policy's parameters;
+            # self.a_grads = the gradients of the policy to get more Q
+            # tf.gradients will calculate dys/dxs with a initial gradients for ys, so this is dq/da * da/dparams
+            self.policy_grads_and_vars = tf.gradients(ys=self.a, xs=self.e_params, grad_ys=a_grads)
+
+        with tf.variable_scope('A_train'):
+            opt = tf.train.RMSPropOptimizer(-self.lr)  # (- learning rate) for ascent policy
+            self.train_op = opt.apply_gradients(zip(self.policy_grads_and_vars, self.e_params), global_step=GLOBAL_STEP)
+
+
+###############################  Critic  ####################################
+
+class Critic(object):
+    def __init__(self, sess, state_dim, action_dim, learning_rate, gamma, t_replace_iter, a, a_):
+        self.sess = sess
+        self.s_dim = state_dim
+        self.a_dim = action_dim
+        self.lr = learning_rate
+        self.gamma = gamma
+        self.t_replace_iter = t_replace_iter
+        self.t_replace_counter = 0
+
+        with tf.variable_scope('Critic'):
+            # Input (s, a), output q
+            self.a = a
+            self.q = self._build_net(S, self.a, 'eval_net', trainable=True)
+
+            # Input (s_, a_), output q_ for q_target
+            self.q_ = self._build_net(S_, a_, 'target_net', trainable=False)    # target_q is based on a_ from Actor's target_net
+
+            self.e_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Critic/eval_net')
+            self.t_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='Critic/target_net')
+
+        with tf.variable_scope('target_q'):
+            self.target_q = R + self.gamma * self.q_
+
+        with tf.variable_scope('abs_TD'):
+            self.abs_td = tf.abs(self.target_q - self.q)
+        self.ISWeights = tf.placeholder(tf.float32, [None, 1], name='IS_weights')
+        with tf.variable_scope('TD_error'):
+            self.loss = tf.reduce_mean(self.ISWeights * tf.squared_difference(self.target_q, self.q))
+
+        with tf.variable_scope('C_train'):
+            self.train_op = tf.train.AdamOptimizer(self.lr).minimize(self.loss, global_step=GLOBAL_STEP)
+
+        with tf.variable_scope('a_grad'):
+            self.a_grads = tf.gradients(self.q, a)[0]   # tensor of gradients of each sample (None, a_dim)
+
+    def _build_net(self, s, a, scope, trainable):
+        with tf.variable_scope(scope):
+            init_w = tf.random_normal_initializer(0., 0.01)
+            init_b = tf.constant_initializer(0.01)
+
+            two = tf.constant(2.0, dtype=tf.float32)
+            def UAF2(x):
+                return tf.nn.relu(x)
+
+
+            with tf.variable_scope('l1'):
+                n_l1 = 700
+                # combine the action and states together in this way
+                w1_s = tf.get_variable('w1_s', [self.s_dim, n_l1], initializer=init_w, trainable=trainable)
+                w1_a = tf.get_variable('w1_a', [self.a_dim, n_l1], initializer=init_w, trainable=trainable)
+                b1 = tf.get_variable('b1', [1, n_l1], initializer=init_b, trainable=trainable)
+                net = UAF2(tf.matmul(s, w1_s) + tf.matmul(a, w1_a) + b1)
+            with tf.variable_scope('l2'):
+                net = tf.layers.dense(net, 20, activation=UAF2, kernel_initializer=init_w,
+                                      bias_initializer=init_b, name='l2', trainable=trainable)
+            with tf.variable_scope('q'):
+                q = tf.layers.dense(net, 1, kernel_initializer=init_w, bias_initializer=init_b, trainable=trainable)   # Q(s,a)
+        return q
+
+    def learn(self, s, a, r, s_, ISW):
+        _, abs_td = self.sess.run([self.train_op, self.abs_td], feed_dict={S: s, self.a: a, R: r, S_: s_, self.ISWeights: ISW})
+        if self.t_replace_counter % self.t_replace_iter == 0:
+            self.sess.run([tf.assign(t, e) for t, e in zip(self.t_params, self.e_params)])
+        self.t_replace_counter += 1
+        return abs_td
+
+
+class SumTree(object):
+    """
+    This SumTree code is modified version and the original code is from:
+    https://github.com/jaara/AI-blog/blob/master/SumTree.py
+
+    Story the data with it priority in tree and data frameworks.
+    """
+    data_pointer = 0
+
+    def __init__(self, capacity):
+        self.capacity = capacity  # for all priority values
+        self.tree = np.zeros(2 * capacity - 1)+1e-5
+        # [--------------Parent nodes-------------][-------leaves to recode priority-------]
+        #             size: capacity - 1                       size: capacity
+        self.data = np.zeros(capacity, dtype=object)  # for all transitions
+        # [--------------data frame-------------]
+        #             size: capacity
+
+    def add_new_priority(self, p, data):
+        leaf_idx = self.data_pointer + self.capacity - 1
+
+        self.data[self.data_pointer] = data  # update data_frame
+        self.update(leaf_idx, p)  # update tree_frame
+        self.data_pointer += 1
+        if self.data_pointer >= self.capacity:  # replace when exceed the capacity
+            self.data_pointer = 0
+
+    def update(self, tree_idx, p):
+        change = p - self.tree[tree_idx]
+
+        self.tree[tree_idx] = p
+        self._propagate_change(tree_idx, change)
+
+    def _propagate_change(self, tree_idx, change):
+        """change the sum of priority value in all parent nodes"""
+        parent_idx = (tree_idx - 1) // 2
+        self.tree[parent_idx] += change
+        if parent_idx != 0:
+            self._propagate_change(parent_idx, change)
+
+    def get_leaf(self, lower_bound):
+        leaf_idx = self._retrieve(lower_bound)  # search the max leaf priority based on the lower_bound
+        data_idx = leaf_idx - self.capacity + 1
+        return [leaf_idx, self.tree[leaf_idx], self.data[data_idx]]
+
+    def _retrieve(self, lower_bound, parent_idx=0):
+        """
+        Tree structure and array storage:
+
+        Tree index:
+             0         -> storing priority sum
+            / \
+          1     2
+         / \   / \
+        3   4 5   6    -> storing priority for transitions
+
+        Array type for storing:
+        [0,1,2,3,4,5,6]
+        """
+        left_child_idx = 2 * parent_idx + 1
+        right_child_idx = left_child_idx + 1
+
+        if left_child_idx >= len(self.tree):  # end search when no more child
+            return parent_idx
+
+        if self.tree[left_child_idx] == self.tree[right_child_idx]:
+            return self._retrieve(lower_bound, np.random.choice([left_child_idx, right_child_idx]))
+        if lower_bound <= self.tree[left_child_idx]:  # downward search, always search for a higher priority node
+            return self._retrieve(lower_bound, left_child_idx)
+        else:
+            return self._retrieve(lower_bound - self.tree[left_child_idx], right_child_idx)
+
+    @property
+    def root_priority(self):
+        return self.tree[0]  # the root
+
+
+class Memory(object):  # stored as ( s, a, r, s_ ) in SumTree
+    """
+    This SumTree code is modified version and the original code is from:
+    https://github.com/jaara/AI-blog/blob/master/Seaquest-DDQN-PER.py
+    """
+    epsilon = 0.001  # small amount to avoid zero priority
+    alpha = 0.6  # [0~1] convert the importance of TD error to priority
+    beta = 0.4  # importance-sampling, from initial value increasing to 1
+    beta_increment_per_sampling = 1e-5  # annealing the bias
+    abs_err_upper = 1   # for stability refer to paper
+
+    def __init__(self, capacity):
+        self.tree = SumTree(capacity)
+
+    def store(self, error, transition):
+        p = self._get_priority(error)
+        self.tree.add_new_priority(p, transition)
+
+    def prio_sample(self, n):
+        batch_idx, batch_memory, ISWeights = [], [], []
+        segment = self.tree.root_priority / n
+        self.beta = np.min([1, self.beta + self.beta_increment_per_sampling])  # max = 1
+
+        min_prob = np.min(self.tree.tree[-self.tree.capacity:]) / self.tree.root_priority
+        maxiwi = np.power(self.tree.capacity * min_prob, -self.beta)  # for later normalizing ISWeights
+        for i in range(n):
+            a = segment * i
+            b = segment * (i + 1)
+            lower_bound = np.random.uniform(a, b)
+            while True:
+                idx, p, data = self.tree.get_leaf(lower_bound)
+                if type(data) is int:
+                    i -= 1
+                    lower_bound = np.random.uniform(segment * i, segment * (i+1))
+                else:
+                    break
+            prob = p / self.tree.root_priority
+            ISWeights.append(self.tree.capacity * prob)
+            batch_idx.append(idx)
+            batch_memory.append(data)
+
+        ISWeights = np.vstack(ISWeights)
+        ISWeights = np.power(ISWeights, -self.beta) / maxiwi  # normalize
+        return batch_idx, np.vstack(batch_memory), ISWeights
+
+    def random_sample(self, n):
+        idx = np.random.randint(0, self.tree.capacity, size=n, dtype=np.int)
+        return np.vstack(self.tree.data[idx])
+
+    def update(self, idx, error):
+        p = self._get_priority(error)
+        self.tree.update(idx, p)
+
+    def _get_priority(self, error):
+        error += self.epsilon   # avoid 0
+        clipped_error = np.clip(error, 0, self.abs_err_upper)
+        return np.power(clipped_error, self.alpha)
+
+
+sess = tf.Session()
+
+# Create actor and critic.
+actor = Actor(sess, ACTION_DIM, ACTION_BOUND, LR_A, REPLACE_ITER_A)
+critic = Critic(sess, STATE_DIM, ACTION_DIM, LR_C, GAMMA, REPLACE_ITER_C, actor.a, actor.a_)
+actor.add_grad_to_graph(critic.a_grads)
+
+M = Memory(MEMORY_CAPACITY)
+
+saver = tf.train.Saver(max_to_keep=100)
+
+if LOAD_MODEL:
+    all_ckpt = tf.train.get_checkpoint_state('./data', 'checkpoint').all_model_checkpoint_paths
+    saver.restore(sess, all_ckpt[-1])
+else:
+    if os.path.isdir(DATA_PATH): shutil.rmtree(DATA_PATH)
+    os.mkdir(DATA_PATH)
+    sess.run(tf.global_variables_initializer())
+
+if OUTPUT_GRAPH:
+    tf.summary.FileWriter('logs', graph=sess.graph)
+
+var = 3  # control exploration
+var_min = 0.01
+
+
+dataarr = []
+
+
+for i_episode in range(MAX_EPISODES):
+    # s = (hull angle speed, angular velocity, horizontal speed, vertical speed, position of joints and joints angular speed, legs contact with ground, and 10 lidar rangefinder measurements.)
+    s = env.reset()
+    ep_r = 0
+    while True:
+        if RENDER:
+            env.render()
+        a = actor.choose_action(s)
+        a = np.clip(np.random.normal(a, var), -1, 1)    # add randomness to action selection for exploration
+        s_, r, done, _ = env.step(a)    # r = total 300+ points up to the far end. If the robot falls, it gets -100.
+
+        if r == -100: r = -2
+        ep_r += r
+
+        transition = np.hstack((s, a, [r], s_))
+        max_p = np.max(M.tree.tree[-M.tree.capacity:])
+        M.store(max_p, transition)
+
+        if GLOBAL_STEP.eval(sess) > MEMORY_CAPACITY/20:
+            var = max([var*0.9999, var_min])  # decay the action randomness
+            tree_idx, b_M, ISWeights = M.prio_sample(BATCH_SIZE)    # for critic update
+            b_s = b_M[:, :STATE_DIM]
+            b_a = b_M[:, STATE_DIM: STATE_DIM + ACTION_DIM]
+            b_r = b_M[:, -STATE_DIM - 1: -STATE_DIM]
+            b_s_ = b_M[:, -STATE_DIM:]
+
+            abs_td = critic.learn(b_s, b_a, b_r, b_s_, ISWeights)
+            actor.learn(b_s)
+            for i in range(len(tree_idx)):  # update priority
+                idx = tree_idx[i]
+                M.update(idx, abs_td[i])
+        if GLOBAL_STEP.eval(sess) % SAVE_MODEL_ITER == 0:
+            ckpt_path = os.path.join(DATA_PATH, 'DDPG.ckpt')
+            save_path = saver.save(sess, ckpt_path, global_step=GLOBAL_STEP, write_meta_graph=False)
+            print("\nSave Model %s\n" % save_path)
+
+        if done:
+            if "running_r" not in globals():
+                running_r = ep_r
+            else:
+                running_r = 0.95*running_r + 0.05*ep_r
+            #if running_r > DISPLAY_THRESHOLD: RENDER = True
+            #else: RENDER = False
+
+            stats = np.array([running_r, env.unwrapped.hull.position[0] ])
+            dataarr.append(stats)
+
+            done = '| Achieve ' if env.unwrapped.hull.position[0] >= END_POINT else '| -----'
+            print('Episode:', i_episode,
+                  done,
+                  '| Running_r: %i' % int(running_r),
+                  '| Epi_r: %.2f' % ep_r,
+                  '| Exploration: %.3f' % var,
+                  '| Pos: %.i' % int(env.unwrapped.hull.position[0]),
+                  '| LR_A: %.6f' % sess.run(LR_A),
+                  '| LR_C: %.6f' % sess.run(LR_C),
+                  )
+            break
+
+        s = s_
+        sess.run(INCREASE_GS)
+
+
+elapsed_time = time.process_time() - st
+
+data_dir = "time_" + str(fold) + ".npy"
+np.save(data_dir, np.array([elapsed_time]))
+
+
+dataarr = np.asarray(dataarr)
+data_dir = "stats_" + str(fold) + ".npy"
+np.save(data_dir, dataarr)

tensorflow/mnist_UAF.py

@@ -0,0 +1,140 @@
+import tensorflow as tf
+print("TensorFlow version:", tf.__version__)
+
+from tensorflow.keras.layers import Dense, Flatten, Conv2D
+from tensorflow.keras import Model
+
+
+
+mnist = tf.keras.datasets.mnist
+
+(x_train, y_train), (x_test, y_test) = mnist.load_data()
+x_train, x_test = x_train / 255.0, x_test / 255.0
+
+# Add a channels dimension
+x_train = x_train[..., tf.newaxis].astype("float32")
+x_test = x_test[..., tf.newaxis].astype("float32")
+
+
+
+train_ds = tf.data.Dataset.from_tensor_slices(
+    (x_train, y_train)).shuffle(10000).batch(32)
+
+test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)
+
+
+
+
+
+class UAF(Model):
+  def __init__(self):
+    super(UAF, self).__init__()
+    self.A = tf.Variable(10.0, trainable=True)  
+    self.B = tf.Variable(0.0000012, trainable=True)   
+    self.C = tf.Variable(0.0000001, trainable=True)  
+    self.D = tf.Variable(9.0, trainable=True) 
+    self.E = tf.Variable(0.00000102, trainable=True)   
+    
+
+  def call(self, input):
+    P1 = (self.A*(input+self.B)) + (self.C *  tf.math.square(input))
+    P2 = (self.D*(input-self.B))
+
+    P3 =  tf.nn.relu(P1) + tf.math.log1p(tf.math.exp(-tf.math.abs(P1)))
+    P4 =  tf.nn.relu(P2) + tf.math.log1p(tf.math.exp(-tf.math.abs(P2)))
+    
+    return P3 - P4  + self.E
+
+
+
+class MyModel(Model):
+  def __init__(self):
+    super(MyModel, self).__init__()
+    self.conv1 = Conv2D(32, 3, activation=None)
+    self.flatten = Flatten()
+    self.d1 = Dense(128, activation=None)
+    self.d2 = Dense(10, activation=None)
+    self.act0 = UAF()
+    self.act1 = UAF()
+    self.act2 = UAF()
+    
+
+  def call(self, x):
+    x = self.conv1(x)
+    x = self.act0(x)
+    x = self.flatten(x)
+    x = self.d1(x)
+    x = self.act1(x)
+    x = self.d2(x)
+    return self.act2(x)
+
+# Create an instance of the model
+model = MyModel()
+
+
+
+
+
+loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
+
+optimizer = tf.keras.optimizers.Adam()
+
+
+
+train_loss = tf.keras.metrics.Mean(name='train_loss')
+train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')
+
+test_loss = tf.keras.metrics.Mean(name='test_loss')
+test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')
+
+
+
+
+@tf.function
+def train_step(images, labels):
+  with tf.GradientTape() as tape:
+    # training=True is only needed if there are layers with different
+    # behavior during training versus inference (e.g. Dropout).
+    predictions = model(images, training=True)
+    loss = loss_object(labels, predictions)
+  gradients = tape.gradient(loss, model.trainable_variables)
+  optimizer.apply_gradients(zip(gradients, model.trainable_variables))
+
+  train_loss(loss)
+  train_accuracy(labels, predictions)
+
+
+@tf.function
+def test_step(images, labels):
+  # training=False is only needed if there are layers with different
+  # behavior during training versus inference (e.g. Dropout).
+  predictions = model(images, training=False)
+  t_loss = loss_object(labels, predictions)
+
+  test_loss(t_loss)
+  test_accuracy(labels, predictions)
+
+
+EPOCHS = 20
+
+for epoch in range(EPOCHS):
+  # Reset the metrics at the start of the next epoch
+  train_loss.reset_states()
+  train_accuracy.reset_states()
+  test_loss.reset_states()
+  test_accuracy.reset_states()
+
+  for images, labels in train_ds:
+    train_step(images, labels)
+
+  for test_images, test_labels in test_ds:
+    test_step(test_images, test_labels)
+
+  print(
+    f'Epoch {epoch + 1}, '
+    f'Loss: {train_loss.result()}, '
+    f'Accuracy: {train_accuracy.result() * 100}, '
+    f'Test Loss: {test_loss.result()}, '
+    f'Test Accuracy: {test_accuracy.result() * 100}'
+  )
+