[ { "path": "table_paper/2407.00101v1.json", "table_id": "1", "section": "7.1", "all_context": [ "Plots 4 and 5 shows the average values of testing accuracy, testing loss, and training loss for five rounds of training from random initialization on the MNIST dataset.", "It can be seen clearly that our algorithm maintains the lead in terms of accuracy and loss as compared to both asynchronous and synchronous versions.", "The same trend is observed for all the combinations of batch sizes and step sizes.", "However, the speed gain by our algorithm over the asynchronous version is not that significant, we believe that MNIST poses a simple optimization problem that does not bring out problems of asynchronous algorithm effectively.", "Table 1 shows the difference of the metrics like accuracy and loss between our algorithm and asynchronous algorithm averaged over the entire training interval.", "For better performance, the difference in accuracy should be positive and that loss should be negative.", "For the next set of experiments, we selected CIFAR-10 as our dataset since we believe that it provides a difficult optimization problem as compared to MNIST.", "Table 2 and plots 6 and 7 show similar statistics as that for MNIST.", "We can clearly note here that our algorithms show significant speedup as compared to both of the other algorithms.", "It is able to achieve higher accuracy and lower loss as compared to asynchronous and synchronous algorithms.", "In all the previous experiments, the synchronous algorithm was very slow, and hence for future analysis, only present a comparison between our algorithm and the asynchronous algorithm.", "" ], "target_context_ids": [ 4, 5 ], "selected_paragraphs": [ "[paragraph id = 4] Table 1 shows the difference of the metrics like accuracy and loss between our algorithm and asynchronous algorithm averaged over the entire training interval.", "[paragraph id = 5] For better performance, the difference in accuracy should be positive and that loss should be negative." ], "table_html": "
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
(300,32)(300,64)(500,32)(500,64)
Test Accuracy1.374-0.5161.3661.291
Test loss-0.0470.001-0.053-0.022
Train loss-0.047-0.001-0.054-0.023
\n
Table 1. Difference between the metric for our algorithm and asynchronous algorithm averaged over entire training interval for MNIST dataset. For better performance, difference in accuracy should be positive and that loss should be negative
\n
", "perturb_sentence_id": [ 4, 5 ], "output": { "perturbed_statement": "[paragraph id = 4] Table 1 shows the difference of the metrics like accuracy and loss between our algorithm and asynchronous algorithm averaged over the entire training interval. For better performance, the difference in accuracy should be negative and that loss should be negative.", "perturbed_explanation": "Original Explanation: The original statement correctly asserts that for better performance, the difference in accuracy should be positive and that in loss should be negative. This implies that our algorithm should exceed the asynchronous algorithm in terms of accuracy, and have a lower loss. 1. The statement is incorrect because it claims that for better performance, the difference in accuracy should be negative. If the difference in accuracy is negative, it would mean that our algorithm performs worse in terms of accuracy compared to the asynchronous algorithm. Thus, a negative difference in accuracy would not indicate better performance." } }, { "path": "table_paper/2407.00101v1.json", "table_id": "2", "section": "7.1", "all_context": [ "Plots 4 and 5 shows the average values of testing accuracy, testing loss, and training loss for five rounds of training from random initialization on the MNIST dataset.", "It can be seen clearly that our algorithm maintains the lead in terms of accuracy and loss as compared to both asynchronous and synchronous versions.", "The same trend is observed for all the combinations of batch sizes and step sizes.", "However, the speed gain by our algorithm over the asynchronous version is not that significant, we believe that MNIST poses a simple optimization problem that does not bring out problems of asynchronous algorithm effectively.", "Table 1 shows the difference of the metrics like accuracy and loss between our algorithm and asynchronous algorithm averaged over the entire training interval.", "For better performance, the difference in accuracy should be positive and that loss should be negative.", "For the next set of experiments, we selected CIFAR-10 as our dataset since we believe that it provides a difficult optimization problem as compared to MNIST.", "Table 2 and plots 6 and 7 show similar statistics as that for MNIST.", "We can clearly note here that our algorithms show significant speedup as compared to both of the other algorithms.", "It is able to achieve higher accuracy and lower loss as compared to asynchronous and synchronous algorithms.", "In all the previous experiments, the synchronous algorithm was very slow, and hence for future analysis, only present a comparison between our algorithm and the asynchronous algorithm.", "" ], "target_context_ids": [ 6, 7, 8, 9 ], "selected_paragraphs": [ "[paragraph id = 6] For the next set of experiments, we selected CIFAR-10 as our dataset since we believe that it provides a difficult optimization problem as compared to MNIST.", "[paragraph id = 7] Table 2 and plots 6 and 7 show similar statistics as that for MNIST.", "[paragraph id = 8] We can clearly note here that our algorithms show significant speedup as compared to both of the other algorithms.", "[paragraph id = 9] It is able to achieve higher accuracy and lower loss as compared to asynchronous and synchronous algorithms." ], "table_html": "
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
(300,32)(300,64)(500,32)(500,64)
Test Accuracy4.8492.4353.4682.884
Test loss-0.137-0.066-0.092-0.080
Train loss-0.139-0.067-0.091-0.082
\n
Table 2. Difference between the metric for our algorithm and asynchronous algorithm averaged over entire training interval for CIFAR-10 dataset. For better performance, difference in accuracy should be positive and that loss should be negative
\n
", "perturb_sentence_id": [ 7, 8 ], "output": { "perturbed_statement": "[paragraph id = 7] Table 2 and plots 6 and 7 show similar statistics as that for CIFAR-10. We can clearly note here that our algorithms show significant slowdown as compared to both of the other algorithms.", "perturbed_explanation": "The original explanation stated: Table 2 and plots 6 and 7 demonstrate statistics in which the algorithms show notable speedup compared to other algorithms, indicating superior performance. 1. Changing 'MNIST' to 'CIFAR-10': The original statement described the statistics as being similar to those for MNIST, providing context for comparison. However, the statement now inaccurately implies the statistics are merely related to CIFAR-10 without juxtaposing this against any other dataset. 2. Changing 'speedup' to 'slowdown': The original explanation highlighted the improved performance of the algorithms through speedup. The statement inaccurately reflects a 'slowdown', which contradicts the achievement of higher accuracy and performance described in the context." } }, { "path": "table_paper/2407.00101v1.json", "table_id": "3", "section": "7.2", "all_context": [ "Further, we wanted to understand how different values of batch sizes affect the efficiency of our approach.", "For each of the batch sizes, we executed 5 rounds of training, each with different initialization of the parameters on the randomly generated dataset.", "Table 3 shows the difference of the metrics like accuracy and loss between our algorithm and asynchronous algorithm averaged over the entire training interval.", "We hypothesized that as the batch size increases, the difference should decrease since asynchronous algorithms start providing updates with high confidence.", "This can be also validated by the trend observed in the plot 8 .", "" ], "target_context_ids": [ 0, 2, 3 ], "selected_paragraphs": [ "[paragraph id = 0] Further, we wanted to understand how different values of batch sizes affect the efficiency of our approach.", "[paragraph id = 2] Table 3 shows the difference of the metrics like accuracy and loss between our algorithm and asynchronous algorithm averaged over the entire training interval.", "[paragraph id = 3] We hypothesized that as the batch size increases, the difference should decrease since asynchronous algorithms start providing updates with high confidence." ], "table_html": "
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
8163264128
Test Accuracy4.8965.1834.2223.3042.599
Test loss-0.141-0.141-0.117-0.089-0.072
Train loss-0.143-0.141-0.114-0.088-0.068
\n
Table 3. Difference between the metric for our algorithm and asynchronous algorithm averaged over entire training interval for various batch sizes and constant step size of 500. For better performance, difference in accuracy should be positive and that loss should be negative
\n
", "perturb_sentence_id": [ 2, 3 ], "output": { "perturbed_statement": "Table 3 shows the effect of batch sizes only on the metrics like accuracy and loss within our algorithm during the initial training phase. We hypothesized that as the batch size increases, the difference should increase since asynchronous algorithms start providing high-volume updates.", "perturbed_explanation": "The original explanation states the comparison between our algorithm and the asynchronous algorithm, hypothesizing that an increase in batch size should decrease the difference between them during the entire training interval, as larger batch sizes in asynchronous algorithms provide updates with higher confidence. 1. The statement incorrectly suggests that Table 3 only shows the effect of batch sizes on the metrics within our algorithm during the initial training phase, rather than comparing it with the asynchronous algorithm and over the entire training interval. 2. The statement incorrectly claims the increase in batch size would increase the difference, whereas it should decrease it, since the asynchronous algorithm provides updates with high confidence as the batch size increases." } } ]